feat(optimize): kind: optimization — factorial/response-surface campaigns

[sriumcp]

Adds kind: optimization — a factorial / response-surface campaign type alongside the default reflective one.

Refs #120. Design spec: docs/superpowers/specs/2026-08-13-optimization-campaign-kind-design.md. Authoring guide: docs/optimization-campaign-guide.md.

Status: end-to-end working and validated by four real campaigns against inference-sim. An earlier revision of this description carried a "this lands the machinery, not a usable command" warning — that is no longer true and the section has been removed. See Validation for what was measured and What running it found for the defects that only real runs surfaced.

Why

The blog post "Discovery Is Easy, Composition Is Hard" benchmarked seven LLM-driven optimizers on the Certus cold-read path. Six beat baseline; five stalled near half the available headroom. The decisive fact was that batching cost 9.5% in isolation yet was required for the winning compound — so any search that prunes regressing candidates cannot reach the optimum.

That is a property of one-factor-at-a-time search, not of the problem. Factorial designs have no composition barrier because they estimate interaction terms directly instead of composing increments. A resolution-V fractional design over 8 factors gets every main effect and all 28 two-factor interactions unconfounded.

It also addresses the cost complaint: the per-configuration decision moves out of the model and into Python.

What lands

Five stages, one per iteration: build → verify → screen → refine → confirm. Only build makes a model call; verify, screen, refine and confirm make zero.

build is opt-in and absent from the default order, so every campaign written before it existed behaves identically. Declare it first when the mechanism under study does not exist in the target yet and the campaign has to author it. Substantive model calls per campaign: ~3 without build, ~4 with it, against 60–90 tokenless benchmark runs.

A new orchestrator/optimize/ subpackage (12 modules, all pure functions over data), four artifact schemas, the optimization campaign block with twelve cross-field validation rules, kind-scoped gate defaults, and nous validate campaign FILE for authoring-time checks.

The architectural guarantee

orchestrator/iteration.py: 23 insertions, 0 deletions — one delegation branch plus one comment. A campaign with no kind, or kind: reflective, behaves exactly as before. tests/test_optimize_no_regression.py proves it structurally (AST-parsed, so formatting changes cannot produce a false pass), including the negative case that complexity_tier remains legal on reflective.

Full suite: 1709 passed, 1 skipped.

Correctness

Every campaign declares two mandatory check families per factor: a manipulation predicate (did the lever engage?) and native property/metamorphic tests in the target's own idiom (is the mechanism correct?). Nous checks the contract; the target's runner produces the verdict, so the harness needs no language knowledge and the tests outlive the campaign.

Relations are classified correctness (hard-fails the campaign) versus behavioral (recorded as a finding, campaign continues). That split is deliberate: a monotonicity break is a discovery — the motivating lever was −9.5% alone and required in combination — so hard-failing on it would make the campaign blind to exactly what it exists to find.

build makes no correctness judgement. verify remains the gate, so the stage that writes the mechanism is never the stage that certifies it, and certification is pure Python reading a real test runner's real output. verify is fail-closed: a declared native_test that did not execute counts as a failure, because "not run" is not evidence of correctness.

Three checks fire regardless of gate approval, since auto-approve is this kind's default: design-matrix fidelity drift, a held-out metric reaching a fitting input, and a correctness-relation violation.

Quality guards on the reported optimum

Three layers, added because a real campaign tripped each one:

1. Hull check. confirm refuses a fitted stationary point whose coded coordinates leave [-1, 1] and replicates the best observed configuration instead. decide_after_refine already detected this and wrote "ranges were too narrow to contain the optimum — escalate before confirming" into findings, but Trigger is reported-not-acted-on, so confirm did it anyway.
2. Best-observed comparison. confirmation.json records confirmed_is_best_observed and, when they disagree, the absolute and percentage gap. "The prediction reproduced" and "this is the best configuration found" are different claims.
3. Diagnostics preservation. Full test output lands in runs/iter-N/test_output.log; a failed configuration's full stdout/stderr lands in runs/iter-N/failed_runs/failed_run_<i>.log with its row index and levels.

On the campaign that motivated these: the fitted optimum was 112.4997 while the campaign had already measured 182.2159 — 38% off, with replicate spread 0.0, reported as CONFIRMED. All three guards now fire on it.

Validation

Four real campaigns against inference-sim, with per-arm git worktrees and an independent third worktree carrying a separate reference implementation for ground truth. Full write-ups in the comments below.

Experiment	Setup	Result
1–2	simulated search strategies, then a target with one live factor	non-discriminating; framing corrected in the comments
3	two real campaigns, existing flags only	optimization complete at $0.082; reflective $5.451 and unfinished — 66×
4	both kinds must author --ceiling-curve in Go first	optimization complete at $2.65, 100% of the true optimum; reflective authored a correct mechanism, then hung on #303 at $6.49

Experiment 4's per-phase split is the durable claim:

	reflective	optimization
one-time authoring	design $1.62	build $2.44
per measurement iteration	execute-analyze $4.77	$0

Authoring code costs roughly the same in both kinds. What this kind removes is the recurring per-iteration measurement cost — 33 real benchmark runs for zero tokens. The advantage therefore scales with how many measurement iterations a question needs, which is why the ratio ranged from 66× (pure measurement) to 2.5× (authoring-dominated).

Honest limits: reflective has never completed in these comparisons (stopped in #3, hung in #4), so every reflective cost figure is a lower bound. And no target has yet shown optimization finding a better answer than reflective on the same question — in experiment 4 it found the joint optimum while reflective isolated a mechanism, a difference in aim rather than in quality.

What running it found

Fourteen defects were found by running real campaigns, none by the test suite — 1600+ tests stayed green throughout, because they inject fakes at exactly the seams that were broken. The full list with consequences and fix commits is in the comments; the ones a reviewer should know about:

Defect	Consequence
test_command had no caller; config_runner never wired	every campaign aborted at verify or screen — the kind was unusable end to end
manipulation assumed targets echo their config back	115 configurations ran, every one failed its check
confirm replicated the geometric origin	found 117.854, reported 73.476, silently
build edited the canonical checkout, not its worktree	two "independent" arms would overwrite each other's mechanism; both trees build, both suites pass
REPORT extractor could not see runs.jsonl	report announced a "storage/persistence gap in the apparatus" while 33 rows of telemetry sat on disk

The last two are worth dwelling on. For the worktree breach, cwd was set correctly, so the code looked right — nothing verified where the edits landed. For the report, _format_results_summary was added by #214 specifically to stop reports dismissing real data as "no data", and it was reintroducing that exact failure for a layout it did not know about.

Process note

Nineteen further defects were found and fixed during implementation, recorded with reasoning in docs/superpowers/plans/2026-08-14-optimization-campaign-kind-EXECUTION-LOG.md. Most were in the plan's own reference code. Three worth a reviewer's attention:

• Two generator entries were labelled resolution V while delivering IV, so a campaign would have reported tight confidence intervals on confounded terms.
• A scalar standard error understated main-effect SEs by up to 14.6%, biasing significance toward false positives — which decides which factors survive screening.
• confirm silently re-ran the screen design while the guide claimed it reproduced the predicted optimum.

Six of the nineteen were in orchestrator/optimize/stage_runner.py, the one module without an independent author, and three consecutive review rounds each found another defect in it on a path no test drove. That file is still the one most worth a human read.

Known limitations

• locked_parameters deviation is not a hard-fail on this path. validate._validate_locked_parameters exists but is reached only from bundle validation, and the matrix path has no bundle.yaml / experiment_spec to compare against. The stage_runner docstring says so rather than claiming a guarantee that does not exist.
• build does not check the spec it is given. It takes the campaign's mechanism description as authoritative and makes one call. In experiment 4 it faithfully implemented a metamorphic ordering I had written backwards; reflective's DESIGN phase caught the error by reasoning about the algebra. An author's algebra error is a blind spot here, which is why the guide now requires a worked numeric example in any metamorphic statement.
• build is one call, not a build-until-green loop. That is deliberate — it bounds cost and stops the model negotiating with its own gate — but it means a verify failure ends the campaign and needs a re-run.

Branch policy: this PR also updates CLAUDE.md and README.md to target main rather than the diverged reflective.

🤖 Generated with Claude Code

Update: compiled experimental policy (paper alignment)

Everything above shipped kind: optimization as a fixed build → verify → screen → refine → confirm schedule. A companion paper (../papers/nousko/paper.tex, "Estimate, Don't Search: Compiled Experimental Policies for Agentic Systems Optimization") describes a stricter version of the same idea: the epoch is not hard-coded control flow but a compiled, pre-registered state machine — policy.json, hashed before the first measurement — interpreted by a pure step() function over a closed observation vocabulary. Measurements choose among registered branches; they never invent one. A semantic exception ends the epoch rather than crossing the boundary implicitly.

Design spec: docs/superpowers/specs/2026-08-16-compiled-policy-design.md (binding authority). Implementation plan: docs/superpowers/plans/2026-08-16-compiled-policy.md (16 tasks, TDD, executed via superpowers:subagent-driven-development — fresh subagent per task, independent task review, controller audit with direct mutation-testing after every task).

Status: all 18 tasks complete (16 planned + 2 inserted mid-plan), plus a final whole-branch review. Phases 0–5 of the plan, the build-stage oracles, systems-target readiness, docs, and a final cross-task review pass are all done. This is the final update to this section.

What Tasks 1–11 closed

Gap vs. the paper (pre-plan)	What now exists
Hard-coded screen → refine → confirm ladder in Python control flow	policy.json: schema-validated, content-hashed, compiled once at verify; step() interprets it. orchestrator/optimize/policy.py.
A stationary point was reported as the optimum, even when it was a saddle or excluded choice factors entirely	decide.recommend() — exact enumeration/argmax over the valid candidate space, no model judgment. Fixes both bugs (Task 7).
No bound of any kind on "how much better could an unmeasured challenger be"	certificate.model_regret_bound / terminal_regret_bound — Bonferroni-corrected simultaneous one-sided bounds, R_δ(x̂) = max_z U_δ(z,x̂), reported honestly as None when no pure-error estimate exists rather than fabricated (Task 8).
Final comparison rested entirely on the fitted model	confirm now measures a shortlist of finalists fresh and compares them independent of the response surface — the paper's terminal discrimination. Full fallback ladder in report.json: certified → terminal_best → model → measured → baseline, plus known_valid_baseline support (Task 9).
Validator only warned about aliasing at low design resolution; no action taken	foldover is a real policy state that spends a measurement block, but only when resolving the alias could change the recommended answer (decide.alias_consequential, quantified over the alternative resolution's ε-optimal set) — the paper's "registered augmentation" (Task 10).
No epoch concept; a semantic failure was an unhandled abort with no answer	A semantic exception now writes epoch_end-<N>.json, and the campaign always returns an action (never an empty/aborted result) even when it hits one. A later fixed re-run recompiles cleanly into a new epoch (Task 11).

Every strict xfail(strict=True) test planted in Task 3 — one per gap above, each naming which future task would flip it — is now a genuine pass. tests/test_optimize_harness.py carries zero xfails for the first time since Task 3. Full suite: 1915 passed, 1 skipped, 0 xfailed (from 1787 at the start of this phase).

Two verified pre-existing defects, fixed early (spec §4)

• D1: every tabulated resolution-IV screen crashed at fit (design matrix is singular) — fit_effects requested one column per two-factor interaction, and at resolution IV aliased columns coincide. Fixed by collapsing to one coefficient per alias class.
• D2: one infeasible row silently NaN-poisoned every fitted coefficient while effects.json stayed schema-valid. Fixed by fitting on the complete-row subset and recording exclusions (fit_exclusions.json).

Genuine judgment calls made during implementation, each independently reviewed and (where warranted) mutation-verified

• confirm's terminal-discrimination shortlist, under model rejection (Task 11). The plan's brief specified that when the fitted model is rejected (lack_of_fit), confirm's shortlist should be built from measured-valid rows only, never the model's ranking. Measured directly: on the bowl and sla oracle surfaces, a strict measured-only rule discards the exact interior optimum refine had already found (because it's a model prediction, not yet a measured row) and, on the constrained surface, caps the achievable answer at 6.12% off truth regardless of how much additional confirm budget is granted — a structural dead end, not a tuning artifact, because a measured-only shortlist can only ever re-measure what's already on disk. The shipped rule instead seeds roughly half the shortlist from measured leaders and fills the rest from the model's ranking — every finalist still gets freshly measured before it can win, so nothing untested is ever returned as the answer. Independent review concluded this is required by the paper's actual constraint (never return an unmeasured configuration — a report-time rule) rather than a shortlist-membership rule, and is correct paper fidelity, not a deviation from it.
• Behind these two calls: a standing project-wide rule, added mid-plan: nous has not reached GA, so until the project owner says otherwise, achieving a piece of work's correct design outranks preserving legacy observable behavior anywhere in the codebase (now CLAUDE.md, top-level, repo-wide — not scoped to this plan or kind). Several of the fixes above changed observable output from what nousko did immediately before this phase; each is named and argued in its task's commit, not silently introduced.

What Tasks 12–16 and the final review closed

Gap vs. the paper (pre-Task-12)	What now exists
No mechanism-drift detection: build could author code and nothing verified the campaign still measured what it claimed to	mechanism.patch / mechanism.sha256 snapshot with hard-fail-on-drift inside the epoch, checked against the compiled policy's own recorded hash (Task 12; scoping fix Tasks 13.5/14.5, below).
A build stage's authored tests could pass trivially (never actually exercising the mechanism) and its control could silently differ from production	Two more build oracles: declared tests must fail before build and pass after, or they prove nothing; known_valid_baseline must equal the pre-build control within tolerance (Task 13).
No common-random-numbers support: a noisy systems target (queueing, caching, autoscaling) needed an order of magnitude more runs to get a usable bound	workload.seed_env / workload.seeds — every row in a comparison gets a deterministic seed, confirm's replicates pair by index across finalists, and terminal_regret_bound computes a tighter paired bound when CRN is genuinely in effect (Task 14).
No worked systems-target campaigns; the adapter contract (run_command/test_command, the seed-honesty check, SLA-as-validity) existed only in scattered docstrings	Three full example campaigns (vLLM batching, Qdrant HNSW, Knative autoscaling) plus docs/targets.md, a ~300-line contract doc with the actual detection recipe for "did the target really read the seed" (Task 15).
The compiled-policy architecture (policy.json, the six states, the fallback ladder, the artifacts an epoch writes) was implemented but not documented anywhere a human or a future agent would read first	A new "The compiled policy" section in the authoring guide, a complete artifact inventory in docs/data-model.md §7, and the policy summary in CLAUDE.md — all describing what the code actually does, not the spec's idealized names (Task 16, below).

Two production defects found live, mid-plan, by running the drift oracle for real (not by the test suite) — this plan's own version of the "run it and see" defects from the section above:

• Task 12's drift oracle hashed the target's entire working tree by content, with no allowlist. Reproduced independently by two reviewers with zero build stage and zero Task 13 code involved: a .pytest_cache/ or run.log left behind by Nous's own pre-existing test-running machinery was enough to trigger a false "mechanism drifted" abort — the worst available misdiagnosis, since it reads as a real problem with the mechanism under study. Fixed via two inserted tasks: Task 13.5 adds an opt-in optimization.build_checks.mechanism_paths allowlist (default unchanged — the whole-tree behavior — so no existing campaign's recorded hash is invalidated); Task 14.5 closes the follow-on gaps (glob-shaped entries silently half-disabling the allowlist; no --smoke-time check that declared paths actually exist). A further gap (normalization-shaped entries like "src//mech.py", and ".." specifically — which makes git exit non-zero and disarms the drift oracle with zero error anywhere) was closed in Task 14's own scope. One residual (absolute paths) was found and carried forward but is lower-risk — an author is far likelier to write a glob than an absolute path.
• A pre-build baseline measurement crashed the entire build-declaring campaign before its one substantive model call ever ran. Task 13's own hoisted baseline_runs call invoked the target's run_command against a tree where the mechanism didn't exist yet — verified by direct A/B execution against the pre-fix commit (mechanism authored: 1 before the regression was introduced, 0/crash after, 1 again post-fix). Fixed by degrading gracefully (a pre_unavailable marker) rather than crashing.

The final whole-branch review

Per-task review is scoped to one task's diff; it structurally cannot see interactions between an early task's assumption and a late task's different one. After all 18 tasks closed, a dedicated final review traced the compiled state machine end-to-end across the whole ~37k-line diff (109 commits) rather than per-task, specifically hunting for that class of gap. It found and fixed four real cross-task defects:

• report.json's recorded path was not epoch-scoped — a campaign whose epoch 1 ended in a semantic exception and recompiled into epoch 2 would report a path that spliced epoch 1's transitions into epoch 2's, naming a state epoch 2 never actually passed through.
• The foldover escape hatch (optimization.policy.foldover: false) was unreachable from any schema-valid campaign — the schema's additionalProperties: false policy block never declared it. Measured the actual cost of the alternative (deleting the dead branch, as an earlier reviewer had recommended) before fixing: on the shipped vLLM example at default resolution, foldover stays registered-but-never-fired and enumerate_paths reports 42 reachable-looking paths through it that the campaign can never take — exactly the harm the hatch exists to prevent. Declared the field in the schema instead of deleting the hatch.
• A test-only harness dataclass held both regret bounds under one ambiguous bare name (residual_regret) that meant the model bound at one stage and the terminal bound at another. Renamed to residual_regret_model (zero external consumers).
• The artifact-inventory table added in Task 16 billed itself as complete but omitted one sibling artifact from the same call site it already documented two others from.

It also found, and the controller fixed directly, one live correctness gap: confirm was the only spending state without a nan_response → exception rule, unlike screen/foldover/refine, which each register the identical rule in the identical position. A NaN response at confirm fell through to confirm's own self-loop default and would run until an outside-the-policy iteration cap cut it off — the one outcome the state machine doesn't register as reachable for a spending state. The fix completes an already-three-times-repeated pattern for the fourth state; mutation-verified (reverting it reproduces the exact self-loop).

Three items were investigated and explicitly left open, tracked rather than force-closed:

• The build oracle's pre/post baseline is measured with no workload.seed_env awareness. Not unsound (pre and post are symmetric — both seedless — so the comparison stays internally consistent) but CRN's variance cancellation is unavailable in a hard-abort gate whose default tolerance isn't meant to police run-to-run noise. A workload-variance-dominated target combined with a build stage could false-abort and blame the mechanism. Recommended as a tracked follow-up issue before any real target combines build with workload.seed_env.
• A handful of OBSERVATION_KEYS have producers but no consuming rule (mostly deliberate/documented; one, behavioral_violation, is fully orphaned).
• No JSON Schema governs report.json/recommendation.json/confirmation.json — the "the two regret bounds must never be collapsed into one number" rule currently rests on a single test assertion rather than a schema constraint.

The branch_id field named in spec §3.9 but never implemented (every transitions.jsonl row carries policy_hash plus the full fired rule instead) was adjudicated as not a real gap — it appears nowhere in the paper itself, and what's on disk is more informative than a bare id would be. Recorded as a spec amendment, not a code change.

Full suite after the final review's fixes: 2009 passed, 1 skipped, 0 xfailed (from 1787 at the start of Phase 1). tests/test_optimize_harness.py — the file that has carried this plan's oracle gate since Task 3 — has carried zero xfails since Task 11 and still does.

Process note

Every task went through the same four-gate pipeline: fresh implementer → independent task review → fix round(s) where warranted → controller-performed audit (scope check, interface check, a mutation the implementer/reviewer hadn't already tried, cumulative full-suite re-verification of every prior task). That audit step caught real issues on its own — including several cases across the plan (Tasks 3, 4, 6, 9) where a controller instruction to an implementer was itself wrong and the implementer's pushback, verified by direct execution, was correct. Two tasks (13.5, 14.5) were inserted mid-plan to close a live production defect discovered organically during implementation, following the same TDD/review/audit process as every planned task. All rulings — every judgment call, every case where a controller instruction was overridden, and their recorded cost-if-wrong — are recorded chronologically in this plan's SDD ledger. Per the workflow, that ledger is deleted at plan completion; the git history, task reports, and this PR description are the durable record.

[sriumcp]

Three commits pushed from a field test of this kind against a real target (BLIS / inference-sim), running kind: optimization and kind: reflective head-to-head on the same problem. Two of them fix defects the field test found in this PR's own machinery; the third documents the apparatus lessons.

fix(optimize): implement apply.kind config_patch, which applied nothing (2c06978)

config_patch was a documented, schema-valid apply kind that was never applied. matrix.py:_render_apply rendered it into a {"patches": [...]} payload and nothing consumed it — runner.py applies only cli_args and env.

Consequence: a campaign whose factors use config_patch runs every row at the baseline configuration while the design matrix, the runs, and the fitted response surface all look real. Neither nous validate campaign (0 errors, 0 warnings) nor --smoke caught it, because the run succeeds — it just measures the wrong thing. Only the run-time manipulation predicate noticed, on row 1 of 18, after a full build stage and ~50 minutes.

Now implemented with RFC 6901 pointers, per-run copies (the author's file is never mutated), a boundary-anchored command rewrite, end-to-end type preservation, and both silent-failure paths closed: validation rule 17 rejects a path absent from run_command, and --smoke verifies each patch actually reached the target.

feat(optimize): optimization.run_timeout_sec (a879fb8)

The run_command timeout was hardcoded to 600 s with no campaign override. This blocks a whole class of objective: one whose evaluation is itself a search. A capacity objective — "highest arrival rate at which the backlog does not grow" — is a bisection over ~5 target runs per point, so per-invocation cost is a multiple of a single run's, and the campaign had no way to say so.

Absent resolves to 600, so no existing campaign changes behaviour. resolve_run_timeout is public because --smoke and the epoch must use the same ceiling — a probe at 600 while the epoch runs at 5400 would turn the one contract check into a source of mismatches.

Also fixes a pre-existing artifact-governance defect found while declaring the field: design_matrix.schema.json has additionalProperties: false but stage_runner was already writing four undeclared fields (policy_hash, workload_seeds, paired, held_fixed). Verified against a real artifact — every campaign with a policy has been emitting a pre-registration record that violates its own schema. Nothing caught it because the schema test only validated a freshly-built payload, never an enriched one. All five now declared, with a test that drives a full synthetic campaign and validates the on-disk artifact. No field became optional.

docs(optimize): §7 pre-flight (0a5b800)

Seven wrong/right pairs, each a defect this field test shipped, following §6's existing convention. Highlights:

• Size run_timeout_sec from the worst corner, not a typical one. Run order is randomized, so the slow corner may run first — you don't get a gentle warning. Timing the baseline (~330 s, comfortably inside 600) then dying on row 1 is exactly what happened.
• --smoke checks that a lever engaged, not that it matters. Eight candidate factors, three unusable: two aborted the target, two were config-captured but consumed by no mechanism.
• Measure the noise floor at the operating point. Apparent −6.8% factor effects in a queue-bound regime became 0.3–3.3% against a 4.8% floor at a healthy one — the large effects were queueing dynamics correlated with the config change.
• Plus: objective fittability (a P99 moving +268% on one factor change while P90 moved +9%; a request deadline pinning three configs at ~300,000 ms), confirming the workload exercises the mechanism (a KV-cache campaign at hit rate exactly 0.000), fail-loud probe harnesses, and pointing certifying relations at tests that fail without the mechanism.

All three: 2085 passed, 1 skipped (from a 2068 baseline).

[sriumcp]

Field test: kind: optimization vs kind: reflective, same task, isolated arms

Both kinds were given the identical task against BLIS (inference-sim) pinned at bb1a5264: implement ARC eviction for the CPU KV-offload tier (which panics today as a declared follow-up), then find the offload configuration that maximizes the sustainable agentic session-arrival rate.

Same workload, hardware, objective, target-side adapter, seeds, and model (claude-opus-5 pinned on every phase for both — reflective's defaults would otherwise have used sonnet for execute_analyze, making a cost comparison meaningless). Isolation was verified rather than assumed: full git clone per arm, per-arm NOUS_CAMPAIGN_PARENT, no shared principles/wiki/run-cache, and a post-hoc grep confirming neither arm's tree mentions the other's run_id. sim/kv/offload_arc.go never appeared in the reflective clone, so neither arm saw the other's implementation.

Axis 1 — cost

	substantive calls	billed
optimization, clean campaign (mechanism exists)	1	$1.68
optimization, first campaign (authors the mechanism)	1	$6.57
optimization, entire run incl. 2 defect-forced restarts	3	$10.21
reflective (1 of 3 iterations completed)	10	$53.09

5.2×–31.7× depending on which comparison is fair. All three are reported; none is the "real" one alone.

The mechanism is not per-turn efficiency — measured, reflective's execute-analyze ran at $0.102/turn, cheaper than build's $0.131. What differs is structural:

• optimization pays one substantive call, then measures tokenlessly. Marginal model cost per benchmark run: $0.
• reflective pays a design + execute_analyze pair per iteration, and those phases can fail expensively: a $9.25 design superseded by a retry, a $13.85 execute_analyze that hit error_max_turns at 121 turns. ~49% of its spend produced no usable result.
• reflective absorbed 10 API-boundary failures (3 SDK-silence aborts, 6 ConnectionRefused, 1 max-turns). The optimization arm had zero — a compiled epoch makes no API calls, so there is nothing to fail. That asymmetry is architectural, not luck.

Axis 2 — correctness: a tie

Both arms independently authored working ARC (Megiddo & Modha four-list, T1/T2 + B1/B2 ghosts). 154/154 and 153/153 of BLIS's own sim/kv tests pass.

The designed discriminator was a stale guard test asserting mustPanic("arc") — implementing ARC makes that assertion false, so an honest arm must update it while a test-defeating one deletes it. Both updated it honestly. Both also wrote an unrequested scan-resistance test that fails if ARC were a no-op wrapper over LRU.

Structural difference only: optimization isolated the mechanism in a new file (357 lines, 178 insertions total); reflective integrated it in place (848 insertions). The trap did not discriminate. With the same strong model on both sides, the kinds were equal here — worth reporting plainly rather than spinning.

Axis 3 — optimality: optimization found a 52% better configuration

	reflective	optimization
valid objective measurements	4	11 (clean screen) / 20 (pooled)
factors varied	eviction_policy × tier-presence	eviction_policy × offload_prompt_only × cpu_bytes × device_class
best λ* found	1.3125	2.0
its conclusion	"ARC ≡ LRU"	device_class dominates; ARC unresolved

Both arms are correct, and the optimization arm found more. Reflective's h_main: REFUTED (ARC ≡ LRU) holds — the optimization data agrees ARC is unresolved (|t| ≈ 0.9). But reflective held offload_prompt_only and cpu_bytes fixed at baseline and never varied device_class at all, so the factor with the largest observed effect was outside its design by construction.

The largest effect is counter-intuitive and has a traced mechanism: sata_ssd beats nvme_gen4 (+0.36 sessions/s, ~28%). cascade() pins each CPU block for its write-through duration, so a slower secondary tier keeps blocks pinned longer, preventing CPU-tier eviction and retaining the hot working set. The slow device acts as an accidental retention policy.

Honest limits: on the clean single-estimator screen (n=11, 4 factors, no replication) no individual factor reaches significance — effect directions are stable but attribution is not established. An earlier |t| = 2.30 for device_class came from pooling rows across two search methods, which inflates n without adding independent information. This screen ranks configurations; confirm is what would certify an effect with a residual-regret bound.

So: optimization wins the deliverable by ~52%, because its factorial design included the factor that mattered — not because it searched better within the same space.

What the run found in this PR's machinery

Seven nous_asks, six fixed in code across the commits on this branch (2140 → passing). Three produced plausible wrong results rather than errors, which is the dangerous class:

• apply.kind: config_patch was schema-documented and applied nothing. Every row of such a campaign ran the baseline while the design matrix, runs, and fitted surface all looked real. validate passed it 0/0; --smoke passed it too, because the run succeeds. Only the run-time manipulation predicate noticed — on row 1 of 18, after a full build stage.
• design_matrix.json never validated against its own schema. Nine undeclared fields including kind: "foldover" — an entire design family the schema rejected. Every campaign reaching confirm emitted a pre-registration record that failed its own schema; the existing test validated a freshly-built skeleton, never the enriched document on disk.
• Rule 12 silently skipped 2 of 3 native_test locator styles, including the bare identifier the Go result parser actually matches on — so it was inert for the style that works.

Plus run_timeout_sec (the hardcoded 600 s ceiling blocked any objective whose evaluation is itself a search), max_parallel (bounded concurrency for confirm replicate blocks only, with screen rows structurally forbidden to opt in), parallel_arms.run_units honoring the max_parallel it previously validated-then-ignored, and --liveness (verify factors matter, and catch a declared level that aborts the target — 3 of 8 candidate factors here were unusable).

The finding I'd most want a reader to take away

My target-side adapter had more defects than Nous did — seven — and they were more expensive. The worst: I edited its output schema three times mid-epoch, so rows measured before each edit carried null for the new keys, and a None reaching float(raw) killed an entire iteration at fit time after ~2 hours of measurement.

Nous already enforces instrument stability for the campaign side: policy.json is content-hashed and _load_or_compile_policy hard-aborts on mismatch. There is no equivalent guard on the adapter — yet the adapter produces every number the pre-registration is about. That asymmetry let me repair the instrument while it ran, three times, and destroy an epoch's comparability with nothing firing.

Guards for this are in progress: an adapter contract hash (fingerprint the response keys+types on the first successful row, hard-abort on drift), output freshness (reject a response byte-identical to the previous row's when levels differ — the defect where a panicking level read as "identical to baseline"), and a declared response.self_check so an author states the invariant their objective must satisfy and Nous enforces it per row. One line —

self_check:
  - {metric: backlog_slope, op: "<=", value: 0.060}

— would have caught 8 of 12 bad rows at the moment each was measured instead of after the epoch. docs/optimization-campaign-guide.md §7 now carries the pre-flight checklist these lessons produced, written target-agnostically.

[sriumcp]

Correction to the field-test comment above — Axis 3 (optimality) restated, not withdrawn

I re-read the reflective arm's findings.json directly instead of trusting my
earlier summary of it. Two factual claims in the comment above are wrong, and
the framing of the optimality axis needs replacing rather than deleting.

The two factual errors

1. "reflective … never varied device_class at all" — false.
Reflective's iteration 2 swept six KV-offload configurations (C0–C5) crossing
eviction policy, offload_prompt_only, CPU tier size, and device class
(nvme vs sata) across three seeds — 54 runs, all present and parsed. I
asserted the opposite twice. My first parse of findings.json used the wrong key
shape, printed nothing, and I read "nothing" as "nothing there" — a silent-empty
parse reads exactly like a null result.

2. "optimization wins the deliverable by ~52%" — not a like-for-like ratio.
The arms searched non-overlapping CPU-tier ranges:

	CPU tier levels swept
reflective	1 GiB, 4 GiB, 16 GiB
optimization	40 GiB, 80 GiB, 160 GiB

No shared level, a 2.5× gap between the ranges. A single ratio across that gap
reads as "better search of one space" when part of it is "a different space."

The objective both arms maximize

Same quantity, same instrument, both arms: λ* — the highest session-arrival
rate sustained without saturating. "Sustained" is a queue-growth criterion, not
a latency threshold: over the arrival-active phase, the OLS slope of in-flight
requests must stay ≤ 0.060 in-flight units per simulated second, and the
last-quartile / first-quartile mean in-flight ratio must stay ≤ 1.40. Either
failing marks the rate "growing." λ* is the longest sustained prefix of the
rate grid rather than max(sustained), so one spurious high verdict cannot
inflate it.

Held identical across arms: BLIS at bb1a5264, agentic multi-turn workload
(70/30 two-cohort, 6 heavy rounds, accumulating context), 8 instances, 300 s
horizon, qwen/qwen3-14b, claude-opus-5 on both sides. The measurement
procedure is identical even where the searched regions differ — that is what
makes the numbers commensurable as measurements.

How the two arms are faring

	reflective	optimization
region searched	CPU tier 1 / 4 / 16 GiB	CPU tier 40 / 80 / 160 GiB
design	6 configs × 3 seeds = 54 runs → 18 cells	4-factor resolution-V screen, 18 rows (15 complete)
best λ* observed	1.3125 (s42, s1337) · 1.59375 (s2024)	2.25 (arc / OPO=False / 40 GiB / nvme_gen4)
spread	identical across all 6 configs, per seed	0.5 → 2.25
conclusion reached	ARC ≡ LRU, with mechanism	DEV/EV directions stable, no factor significant at n=11

Each did what its kind is for, and the two outputs are different in kind:

• Reflective produced a causal explanation with a confirmed prediction. Its
  REFUTED is not a bare null. It identified why the knob is inert:
  mirrorSkipped = 56,435,555 MirrorToCPU stores refused because the CPU tier was
  full and every block was pinned by the write-through cascade — the evictable
  pool is degenerate, so eviction policy cannot matter at that tier size. It
  established ARC was live via an instrumented build (per-instance-varying T1/T2
  splits, reverted before any measurement cell ran) and showed the instrument was
  sensitive (seed moves λ*, policy does not). It then predicted the null would
  hold across all six configs and confirmed it 18/18. A factorial screen
  cannot produce that.
• Optimization produced a ranked surface over a wider space, reached a higher
  λ*, and did it for less. Declaring a 4-factor space up front and sweeping it
  systematically is itself a capability of the kind — the opportunity to
  navigate a designed space is part of what is being compared, not a confound to
  be normalized away.

Whether 2.25 reflects better navigation or a friendlier starting neighborhood is
underdetermined by this data. Both arms may be right about their own regime:
reflective explained why the factor is inert where the tier is starved;
optimization is measuring where it is generous.

Honest limits on the optimization number. Each of the 15 complete screen rows
ran on a different workload seed (31676–31692), so the 0.5→2.25 spread mixes
configuration effect with seed effect — and seed alone is known to move λ* (it
moved reflective's from 1.3125 to 1.59375). 4 rows came back non-monotone. On the
clean n=11 single-estimator screen no individual factor reaches significance;
an earlier |t| = 2.30 for device_class came from pooling rows across two search
methods, which inflates n without adding independent information. Separating
configuration from seed is exactly what confirm's replication does, and it is
what would attach a residual-regret bound to the recommendation.

What is unchanged

• Token budget — optimization, 5.2×–31.7× ($1.68 / $6.57 / $10.21 vs
  $53.09). ~49% of reflective's spend was waste ($9.25 superseded design, $13.85
  hitting error_max_turns at 121 turns); 10 API failures on the reflective arm,
  zero on the optimization arm.
• Correctness — tie. 154/154 and 153/153; both updated the stale
  mustPanic("arc") guard honestly; both wrote an unrequested scan-resistance
  test.

Field-test caveats worth recording

The CPU-level mismatch was my authoring choice, not a property of either
kind — if the goal had been a controlled optimality comparison, the levels
belonged in one shared block both YAMLs reference. Separately, across this field
test my adapter produced eight defects and Nous six, and the two most
expensive failures were both mine (an epoch voided by an and/or inversion; an
iteration killed at fit time by a mid-epoch output-schema edit). Three of those
adapter defects are now structurally impossible to repeat silently — that is what
the adapter guards in 45a9087 are for. Those guards were not armed on this
epoch, which launched before they were committed.

[sriumcp]

Field test: kind: optimization vs kind: reflective on 3b1b/manim

Three arms built the same missing mechanism (dirty tracking in manim's wgpu
draw path) against the same objective, the same shared adapter, the same pinned
commit 9d57bcf, and the same model (claude-opus-5 in every phase, so no cost
figure is a model-price artifact).

arm	kind	what it received
opt	optimization	the campaign as first authored
optp	optimization	parity re-run — identical campaign, but the build received the same facts refl got, after 4a39138 fixed the channel
refl	reflective	4 iterations, its own DESIGN phase each time

Why optp exists — a confound I introduced, then measured

opt's build never received the target's cost facts. I had written them, into
optimization.guidance.factor_nomination. build_prompt never read that field
— the string "guidance" appeared zero times in all of orchestrator/optimize/, and
the guide documented it as "reserved, not read by any stage." refl carried the
same facts in research_question, which is its prompt.

Measured asymmetry: opt's authoring text 1,445 chars, refl's 4,130 (2.86×). Of 13
measured facts, opt's build received 3 and refl's received all 13.

optp is the same campaign re-run with that channel fixed and 0 fact
asymmetries (verified by rendering the real build_prompt and diffing).

---

(a) Cost

arm	USD	calls	model time	output tokens	cache reads	$/measurement
opt	$7.30	8	28.8 min	105K	5.2M	$0.059 (124 runs)
optp	$12.98	5	44.0 min	140K	13.0M	$0.481 (27 runs)
refl	$71.23	26	271.5 min	700K	67.6M	$0.250 (285 runs)

refl spent 5.5× optp and 9.8× opt. Its design phase alone ($27.16) cost
more than both optimization arms combined. Both optimization arms spent 100% of
their tokens in build — every other state is tokenless, which is the kind's
structural property, not an accident of this campaign.

(b) Correctness

arm	tests	tree diff	test lines deleted	verdict
opt	21 pass (from 9)	3 files, +294	0	clean
optp	13 pass	9 files, +534	12	net-positive refactor
refl	53 pass	9 files, +636	12	net-positive refactor

The 12 "deleted" lines in optp and refl are both the same change to
tests/render_compare.py: generalising the golden-image harness's config writer
from booleans-only to typed values, and adding flags to drive the new knobs. I read
every line — no guard was removed. Falsifiability verified in refl's tree by
injecting a real pixel defect: 7 tests fail, including its own new
test_dirty_tracking_is_invisible[4]/[16].

(c) Optimality — one interleaved sweep, idle machine

6 rounds, oracle → opt → optp → refl within each round, each arm at its own
recommendation in its own tree. Interleaved because block ordering fabricated a
result twice in this experiment: a 2.5% phantom gap between byte-identical
trees, and a sign flip on opt's optimality.

Ground truth from a pristine third clone: baseline 0.22998 s, floor
0.09825 s → 57.3% headroom.

arm	recommendation	mean	vs baseline	paired t (df=5)	headroom captured
opt	DT=false (mechanism off)	0.23149 s	−0.66%	−1.76	−1.1%
optp	DT=true, SG=mobject, RI=16	0.22592 s	+1.76%	+3.18	+3.1%
refl	DT=true, SG=mobject, RI=16	0.19351 s	+15.86%	+30.93	+27.7%

Note opt's and refl's recommendations are identical configurations — the 15.9%
gap is entirely in the implementation, not the settings.

Certification

arm	recommendation.basis	R_terminal	ε
opt	terminal_best	null (not estimable)	0.0180
optp	certified	0.00304	0.0180
refl	n/a — the reflective kind has no certificate

Both bounds came back null for opt rather than 0.0 — "not estimable" rather
than a fabricated guarantee. That invariant held.

(d) Wall clock

arm	model time	measurement runs	campaign elapsed
opt	28.8 min	124	~50 min (2 epochs)
optp	44.0 min	27	~50 min
refl	271.5 min	285	~195 min (4 iterations)

Arms ran concurrently and contend for one GPU (+5% at 2-way, +21% at 8-way), so
wall clock is the least clean axis. Model and measurement time are reported
separately because only the latter is affected.

---

What the comparison actually shows

1. The guidance gap was real, and fixing it flipped the sign. Same target,
same knobs, same objective: opt's mechanism was a 23.7% regression (t=25);
optp's was certified with DT: true. That delta is attributable to a plumbing
defect of mine, now fixed in 4a39138.

2. It was not the whole story. At the same config, optp captures 3.1% of the
headroom and refl 27.7%. Both agents had identical facts. The difference is
architectural — measured at subtree/16:

arm	write_uniforms removed	claim removed	timing
opt	70%	47%	+23.7% slower
optp	69%	34%	+3.65%
refl	93%	93%	−10.4%

opt decided by rebuilding a state tuple over every family member each frame —
an O(N) check to avoid O(N) work. refl pushed invalidation to the source (a new
structure_epoch.py, note_structure_changed() from the mutators) so a frame
reads one counter, and added SharedBuffer.restore_used() to replay claims in one
step instead of 57,600 calls.

3. refl decided this at DESIGN time, in one iteration — it did not iterate out
of opt's failure mode. Its bundle.yaml, written before any code, already named
opt's exact defect as a branch to avoid: "If C2 is SLOWER, the epoch check is
being paid on top of the old path rather than instead of it."

4. And it kept improving. Iteration 4 found an apparatus defect in its own
mechanism via preflight (a missing note_flags_changed() call), fixed it, then
measured a further 2.9% (Wilcoxon p=0.000488, 11/12 seeds).

The structural finding, stated carefully

kind: optimization spends one build call; every later state is tokenless.
When that call produces a sound-but-slow design, the kind has no path to revise it
— it can only measure the regression accurately, which both optimization arms
did correctly. opt recommended turning its own mechanism off; optp certified a
real if modest gain. Neither overclaimed.

kind: reflective deliberates implementation strategy in DESIGN before writing
code, and iterates. That cost 5.5–9.8× more and bought 9–25× the headroom here.

This is what motivated the new plan stage (9cae1b8, 9595d7b): one pre-epoch
call that prices the mechanism before build authors it, capturing the ~13% of
refl's design bundle that mattered (87% was experiment design this kind already
pre-registers and hashes — a strictly stronger guarantee). The epoch stays
tokenless; substantive calls go 1 → 2, not 26.

plan is untested in the field. Its value is a hypothesis until an opt run
uses it. The natural next experiment is a fourth arm with plan enabled and the
same parity facts.

Defects found in nous by this field test

#	defect	status
1	guidance.factor_nomination reached no prompt; guide said "reserved"	fixed 4a39138
2	build prompt had no performance requirement, and told the agent probes "buy no measurement"	fixed 4a39138 / 1d8f428
3	response.constraints never reached the build	fixed de76fc7
4	the plan was write-only — nothing checked it against screen	fixed 9595d7b
5	refl's best_found.json scored every candidate 0.0; findings carry empty metric dicts, so its own optimality ranking never populated	open

Five further defects were mine, in the campaign or adapter — --json-report
running zero tests, native_test locators, type: numeric on an integer knob
(which cost two epochs), an inverted guard, and two verifications that could not
fail. All are documented in docs/AUTHORING_DEFECTS.md and docs/RUN_LOG.md in
the field-test workspace.

One caveat on attribution: I authored all three campaigns. Some of the opt→optp
delta is my authoring rather than the kind, and I have not separated those cleanly.
The optp→refl delta is the one I'd defend as structural.