Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
9671a3a
chore(agents): make test skills language-neutral
CBenoit Oct 20, 2026
9d46c40
chore(agents): tighten language-neutral test skills
CBenoit Oct 20, 2026
3be625c
chore(agents): close gaps in the test-pruner fixes
CBenoit Oct 20, 2026
e065236
chore(agents): key Stryker kills by test id
CBenoit Oct 20, 2026
645d498
chore(agents): pass the solution for .NET Framework Stryker runs
CBenoit Oct 21, 2026
eab5674
chore(agents): move Rust test forms into a reference
CBenoit Oct 21, 2026
7e593a5
chore(agents): pin CsCheck for .NET Framework test projects
CBenoit Oct 21, 2026
814d1ef
chore(agents): guard C# pruning against an empty mutant sample
CBenoit Oct 21, 2026
3a69f31
chore(agents): settle test-pruner modes, guards, and triage
CBenoit Oct 21, 2026
852cffe
chore(agents): state the fuzz fallback once for every language
CBenoit Oct 21, 2026
2ce6efb
chore(agents): mark where each test's expected value comes from
CBenoit Oct 21, 2026
34a68c9
chore(agents): keep a protected test's scaffold marker
CBenoit Oct 21, 2026
4a02122
Merge remote-tracking branch 'origin/master' into cbenoit-language-ne…
CBenoit Oct 21, 2026
85846c9
chore(agents): cite scoped contract files in oracle markers
CBenoit Oct 21, 2026
e9bb9d8
chore(agents): merge no scaffold test
CBenoit Oct 21, 2026
ad19e7b
chore(agents): give oracle marker paths from the repository root
CBenoit Oct 21, 2026
e402c4c
chore(agents): spell out regression marker references
CBenoit Oct 21, 2026
d0e9d2f
chore(agents): restrict property markers to named laws
CBenoit Oct 21, 2026
2e4303c
chore(agents): cite files in other repositories by commit
CBenoit Oct 21, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 43 additions & 5 deletions .agents/skills/test-oracle-reviewer/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
---
name: test-oracle-reviewer
description: Judge whether a test's expected value is justified independently of the code under test. Use when reviewing tests added or changed in a pull request, auditing a module's tests, or whenever expected bytes, snapshots, or mock values may have been copied from the implementation's own output.
description: Judge whether a test's expected value is justified independently of the code under test, and define the `// oracle:` marker that records its source. Use when reviewing tests added or changed in a pull request, auditing a module's tests, marking a test's source, or whenever expected bytes, snapshots, or mock values may have been copied from the implementation's own output.
---

# Test oracle reviewer

A test whose expected value came from the code under test detects change, not bugs.
For each test, ask whether the expected value can be justified without running the code.

Independent sources: a spec example, an external capture, a hand-derived value, an input the test chose itself and asserts was honored, a real property such as round-trip or `encode` writing exactly `size()` bytes, or a reference model.
Independent sources: a spec example, an external capture, a hand-derived value, an input the test chose itself and asserts was honored, a named law such as round-trip, a property a document states, such as `encode` writing exactly `size()` bytes, or a reference model.
Self-derived sources: a snapshot of current output, a value computed through the code or a shared helper, a production constant or message copied into the test, a mock that returns what is then asserted, `is_ok()` or `!is_empty()` on an input that no requirement says must succeed, or a restatement of what a derive or the type system guarantees.

Verify instead of guessing.
Expand All @@ -17,10 +17,48 @@ A value pasted from the code's output is independent once it checks out against
A search-engine or AI summary is not a source; cite the primary document.

Classify each test as independent, self-derived, or unknown, with evidence, and report per assertion when a test mixes classes.
When a value checks out, add a one-line citation comment to the test, such as `// <spec> §<section> example` or `// hand-derived; see PR #N`, so the next review is cheap.
When a value checks out, add its marker, as below, so the next review is cheap.
When a value is wrong, report a bug; do not adjust the value or the code.
Unknown is an outcome, not a guess in either direction.

In a pull request review, comment on each self-derived test with the independent source to use and leave the fix to the author.
## Markers

Mark where an expected value comes from with `// oracle: <kind> <reference>` on the test, on a case row, or on one assertion when a test mixes sources, so one search finds every test that rests on a source.
Strong kinds name an independent source, and a test, case, or assertion marked with one is deleted only when its source changes or is shown not to decide what it asserts:

- `spec <document> §<section>`: normative text other than a contract or intent file, such as an RFC, a Microsoft open specification, or `docs/JMUX-spec.md`.
- `contract <path> §<section>`: a `CONTRACT*.md`.
- `intent <path> §<section>`: a statement in an intent file.
- `regression <issue or crash file>`: a bug report, as a Jira key such as `DGW-123` or a GitHub issue such as `#123` or `owner/repo#123`, or a fuzz crash file under `fuzz/`.
- `reference <implementation, capture, or model>`: what another implementation or a reference model does, recorded or compared live.
- `property <law>`: a law of the code under test named by one of `round-trip`, `idempotent`, `involution`, `commutative`, `associative`, or `monotonic`.

Write every path from the repository root, with forward slashes and exact case, such as `crates/gateway-db/INTENT.md`, because markers sit at every depth and a root-relative path reads and resolves the same from each.
A file in another repository is cited as `<owner>/<repo>@<sha>:<path>`, with `<path>` from that repository's root, such as `Devolutions/devolutions-gateway@43958682a1b2:docs/agent-identity/CONTRACT.agent-identity.md`, and `<sha>` a commit of at least 12 hex digits, never a branch or tag, because a pinned commit never changes and so resolves the same every time.
A `spec` document outside `docs/`, such as `RFC 6455` or `MS-RDPBCGR`, is named rather than given as a path.
`§<section>` names the `id` of an anchor the file places before a heading as `<a id="…"></a>`, or else a heading, in any case, or the number a heading starts with, so `§3.2` matches `3.2 Framing`.
Cite the anchor when the file has one, because it survives a renamed heading or a renumbered section.
When a pinned section changes upstream, re-check the test against the new text, then bump the pin or change the test, never the expected value alone, because a value changed without its pin cites text that decides the old one.

A `property` law is accepted only by its well-known name, because the name is its definition, while any other property needs a written source: it takes the kind of the document that states it, and is `scaffold` until one does.
An assertion whose expected value comes from running the code under test is self-derived even over generated inputs, so it is `mutant` or `scaffold`, never `property`.
A codec's `round-trip` test never replaces its spec or capture vectors, because a round-trip passes symmetric bugs.
A hand-derived value, a checked snapshot, or an input the test chose and asserts was honored takes the kind of the source that requires it.

Weak kinds name a self-derived source, which detects change rather than bugs, so any intentional change, such as a refactor or a new feature, may delete a test, case, or assertion marked with one without justification unless `test-pruner` protects it:

- `mutant <faults in words>`: a test or assertion kept to catch mutants that no strong source decides.
- `scaffold`: a test or assertion written for feedback while implementing that no strong source decides.

No `scaffold` marker merges: before the pull request does, give what it marks a strong source, keep it as `mutant` when `test-pruner` finds it alone catches a fault, or delete it.
When it checks a property or intention no document states, ask whether and where to specify it, and mark it with that source once specified, since a confirmation alone is not a source; a protected test takes this path rather than deletion.

Mark a strong source whenever one decides the value, so a test found through a mutant but decided by a spec is `spec`, because whether a test may go depends on whether a source still requires it, not on how it was found.
An unmarked test starts as unknown: verify it as above, and turn a citation in another form into the marker it supports.
A marker is a claim: verify it like any other origin.

## Review

In a pull request review, comment on each self-derived test that an independent source decides, naming that source, and on each changed test without a marker; leave the fix to the author.
When that source needs a small non-test change, such as an injectable deadline the test can choose, request the change.
For mutation evidence on the same pull request, use `test-pruner` in pull request mode.
For mutation evidence on the same pull request, use `test-pruner` in review mode.
52 changes: 31 additions & 21 deletions .agents/skills/test-pruner/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,65 +1,75 @@
---
name: test-pruner
description: Prune, strengthen, and consolidate tests using oracle classification and cargo-mutants evidence, for one Rust crate or module or for a pull request's diff. Use when the user asks to remove redundant tests, dedupe or consolidate tests with rstest or proptest, audit test value, clean up AI-written tests, or get a mutation-backed review of a pull request's tests. For a quick oracle-only check of a pull request's tests, use `test-oracle-reviewer` instead.
description: Prune, strengthen, and consolidate tests using oracle classification and mutation evidence, cargo-mutants for Rust and Stryker.NET for C#, for one crate, project, or module or for one change's diff. Use when asked to remove redundant tests, consolidate tests into cases or properties, audit test value, clean up AI-written tests, finalize a change's tests before its pull request opens, or get a mutation-backed review of a pull request's tests. For a quick oracle-only check, use `test-oracle-reviewer` instead.
---

# Test pruner

Scope each run to one crate or module, or to one pull request's diff, so mutation runs stay affordable and results stay reviewable.
Scope each run to one crate, project, or module, or to one change's diff, so mutation runs stay affordable and results stay reviewable.
Decide from two independent measurements: whether the expected value is justified independently of the code, and which faults only this test catches.
Mutation testing alone cannot tell a test that matches itself from a correct one, and oracle review alone cannot tell which tests are the sole guard for a fault.
Tool mechanics: `references/cargo-mutants.md` for Rust, `references/stryker.md` for C#; load only the language in scope.

Editing mode is for tests you own, such as a change's diff before its pull request opens: commit the changes yourself and hand the evidence tables (oracle table, kill matrix, challenge results, survivors, and found bugs) to whoever posts them as a pull request comment.
Review mode is for another's pull request: post a review and change nothing in its branch, applying mutants and challenge patches only in the clone below.
Run mutants in a clone of their own and address its files by absolute path, since a relative path resolves against whichever checkout the shell starts in, and an applied mutant left in a shared checkout corrupts every other run there.

## Measure

Invoke `test-oracle-reviewer` on every test in scope and record independent, self-derived, or unknown with evidence.
Run cargo-mutants on the scoped source as described in `references/mutants.md`, adding domain mutants from its catalogs because generic mutants miss most domain faults.
Build the kill matrix of mutant by test from the cargo-mutants logs; it needs no per-test reruns.
Run the mutation tool as the reference describes, adding domain mutants from `references/domain-mutants.md` because generic mutants miss most domain faults.
Build the kill matrix of mutant by test from the tool's outputs; only the cases the reference names need per-test reruns.
A unique kill is a mutant caught by exactly one test in scope.
Make no deletion decision about a test until some mutant of the code it exercises is caught, by it or another test, since zero unique kills over nothing caught measures nothing and the invariant then passes trivially; challenge such a test instead.
For tests with several assertions, record which assertion fails for each kill.
Run the scoped suite repeatedly on the clean tree, for example 15 times, to find flaky tests before trusting any kill.

## Decide

Keep independent tests; merge them into rstest cases only when they check the same property and differ only in data.
Keep independent tests; merge them into cases of one parametrized test only when they check the same property and differ only in data.
Never delete an independent test on mutant evidence: the mutant sample is tiny next to the real fault space, and a spec vector costs nothing to keep.
Strengthen a self-derived test with unique kills by replacing its expected value with an independent one; if none exists, keep it and mark it `// oracle: self-derived; sole guard for <mutants>`.
A strengthened test that fails is a finding, never a task to make it pass.
Strengthen a self-derived test with unique kills by replacing its expected value with an independent one; if none exists, keep it with a `mutant` marker, as `test-oracle-reviewer` defines, naming each fault it alone catches in words, since mutant ids and line numbers change between runs.
A strengthened or gap test that fails is a finding, never a task to make it pass: report the test, the independent expected value and its source, and the code's actual output to whoever assigned the run in editing mode, or to the author in review mode, and change neither test nor code meanwhile, since whichever is edited first becomes the specification.
Delete a self-derived test with no unique kills.
A test of unknown oracle with unique kills has already passed the challenge; strengthen it.
Challenge a test of unknown oracle without unique kills.
Challenge a test of unknown oracle without unique kills, unless it is protected below, since its fate does not depend on the answer.
No `scaffold` marker outlives the pass: resolve each as `test-oracle-reviewer` requires, sending its questions to whoever assigned the run, or to the author in review mode.

Deletion is a set decision, not a per-test verdict.
Two tests that both catch mutant M each have zero unique kills, and deleting both loses M.
Choose the deletion set so the caught mutants afterward equal the caught mutants before.
The same rule applies to assertions: a self-derived one that never fails across its test's kills is a deletion candidate.

Never propose deleting a regression test tied to an issue or fuzz crash, a test of untrusted-input parsing or bounds checks, or a doctest; strengthen or merge those only.
Treat a suite that deliberately uses snapshots as a change detector: dedupe it, do not delete it.
Protected tests are never proposed for deletion: a regression test tied to an issue or fuzz crash, a test of untrusted-input parsing or bounds checks, and a doctest.
That holds whatever their marker, because a refactor of a parser is where a dropped bounds check goes unnoticed.
Treat a suite that deliberately uses snapshots as a change detector as protected: dedupe it, do not delete it.
Report flaky tests separately; deleting one hides a bug.

## Challenge

Give a fresh-context agent the test, the code under test, and the mutants the rest of the suite already catches, and ask for one mutant that this test catches and the others miss, as a patch only.
Apply and run the patch yourself; arguments do not count.
Add any mutant it finds to the domain catalog regardless of the verdict.
Record any mutant it finds in the evidence regardless of the verdict, and propose a reusable one for `references/domain-mutants.md` in a separate pull request, since neither mode edits skills.
If it finds none, ask the human who wrote or prompted the test for the source of its expected value, and accept a source you can check, never a confirmation.
Without one, delete the test, or hand a protected test to the file's CODEOWNERS.
Without one, delete the test.

## Consolidate

Keep rstest case names readable, with `#[case::name]` or enum-valued `#[values]`.
Use proptest only for a real property; an assertion that re-implements the unit is a self-derived test with random inputs.
Consolidate into the case and property forms `test-writer` gives for the language, naming every case, since the kill matrix reports each case by its name.
Use a property test only for a real property; an assertion that re-implements the unit is a self-derived test with random inputs.
Round-trip passes symmetric bugs, so every codec module keeps spec or capture vectors.
Do not add a never-panics property where a fuzz target already covers it.
Do not add a never-panics or never-throws property where a fuzz target already covers it.

## Invariant and output

The caught-mutant set never shrinks except through deletions a human approves in a pull request.
Rerun the caught set after every deletion set and every consolidation, verify by rerunning rather than reasoning, and revert on any loss.
Triage surviving mutants as equivalent, coverage gap, or bug lead; a mutant that survives because its code is unreachable is a bug lead.
Rerun the caught set after every deletion set and every consolidation, verify by rerunning rather than reasoning, and revert on any loss; compare runs made with identical tool settings over the same scope.
After ruling out the scoping errors and tool artifacts the reference names, triage each survivor as equivalent (no observable change), unconstrained (an observable change no independent requirement decides, such as a log message's wording), coverage gap (a change an independent requirement rejects), or bug lead (the original code looks wrong, or is unreachable in every supported configuration).
Close each coverage gap with a test at the seam `tdd` defines, marked with the requirement that rejects the change and written with `test-writer`; a survivor only a test of internals could catch is not a gap but equivalent or a bug lead, as above.
A suite of independent tests yields few deletions; a pass may end in consolidations and gap tests only, or in no change.

In crate mode, open separate pull requests for deletions, consolidations, and strengthenings, or separate commits when amending an open pull request; they are different reviews.
Put the oracle table, kill matrix, challenge results, survivors, and found bugs in a pull request comment; the pull request body is the squash commit.
In editing mode, commit deletions, consolidations, strengthenings, and gap tests separately, each as soon as it is verified, so an interrupted run loses little; they are different reviews.
Change only tests and source comments; never modify non-test source otherwise.

In pull request mode, post one review instead: inline comments on the affected tests, with the tables in `<details>` blocks.
Deletions, consolidations, strengthenings, and small non-test changes that enable an independent oracle become requests to the author, with the current caught set as the bar.
In review mode, post one review: inline comments on the affected tests, with the tables in `<details>` blocks.
Deletions, consolidations, strengthenings, gap tests, and small non-test changes that enable an independent oracle become requests to the author, with the current caught set as the bar.
Loading
Loading