You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
chore(agents): make test skills language-neutral - #2136
The test skills stay one skill per job rather than gaining a copy per language, because the method is identical in Rust and C# and only the mechanics differ.
Each test, case, or assertion carries an // oracle: marker naming the source of its expected value. Tests added to catch a mutant are marked weak, and any intentional change may delete them without justification, because coverage driven by mutants rewards tests that pin how the code works today; tests test-pruner protects stay protected whatever their marker. Tests written only for feedback while implementing never merge: they gain a source or are deleted first.
The test skills stay one skill per job rather than gaining a copy per
language, because the method is identical in Rust and C# and only the
mechanics differ; each language's tool mechanics and test forms sit in a
reference loaded only for that language.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Point consolidation forms at `test-writer` instead of restating them in
each mechanics reference, and state the failed-strengthening rule once
on top of `test-oracle-reviewer`. Exempting protected tests from the
challenge leaves no protected test to hand to CODEOWNERS, so that branch
goes, and "protected" is now defined where the tests are listed.
xUnit v2 turns a whole theory into one test case when any row holds a
value it cannot serialize, so `[MemberData]` rows hold only serializable
values. Stryker reruns compare `--mutate` runs, because a test edit
changes which mutants `--since` selects. Domain mutant patches are taken
with `git diff --output`, since a Windows PowerShell redirect writes
UTF-16 with CRLF that `git apply` rejects, and a patch that does not
build proves nothing.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A failing gap test is a finding like a failing strengthened test, and
the rule keeps its reason because it is the one most tempting to break.
Protected tests are no longer offered for merging, since issue
regression tests stay standalone under `test-writer`. The evidence
tables are defined where editing mode first names them, Stryker reruns
for a diff are measured with `--mutate` from the start, and a
challenger's hand-written patch is rebuilt with `git diff --output`.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The Stryker kill matrix now prints each killing test's id beside its
display name, since truncated display names collide. Mutants that share
file, mutator, replacement, and line text are matched by their order in
the file, which test and comment edits leave unchanged. A CsCheck
counterexample is pinned in a regression test rather than an
`[InlineData]` row, so regression tests stay standalone.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The command lacks the solution path required for .NET Framework projects. package/AgentWindowsManaged.Tests/DevolutionsAgent.Installer.Tests.csproj:3 targets net48, so this workflow cannot start mutation testing there unless a solution path is already configured. The Stryker.NET 5.0.0 configuration reference requires it. Document adding --solution <path-to-solution.sln> for these projects.
Stryker.NET cannot run on a .NET Framework test project without a
solution path, and the Agent installer tests target net48.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
`test-writer` now states each rule once without naming a library, and
the Rust forms sit in `references/rust.md` beside the C# ones, so neither
language reads the other's forms and each reference stands alone.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The installer tests target net48 (package/AgentWindowsManaged.Tests/DevolutionsAgent.Installer.Tests.csproj:3), while the upstream CsCheck project targets modern .NET. This guidance does not identify a compatible version for those tests, so following it with an incompatible release would fail package restore. Document a compatible CsCheck version or qualify the recommendation with an alternative for unsupported frameworks.
CsCheck releases after 2.14.1 need .NET 6 or later, and the Agent
installer tests target net48.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🟢 Approval recommended
The documentation-only changes have no blocking issues; the remaining feedback is a minor safeguard clarification.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Apply viable-mutant safeguard to shared measurement step
.agents/skills/test-pruner/SKILL.md:21
The safeguard against an empty or near-empty viable-mutant sample is only in references/cargo-mutants.md:11, which C# runs do not load. With no viable mutants, every self-derived test has zero unique kills, so deleting unprotected tests still preserves the empty caught set. Add the safeguard to this shared measurement step so it also applies to C# pruning.
The rule against deciding deletions on an empty or near-empty viable
sample applies to every mutation tool, so it moves from the
cargo-mutants reference into the shared measurement step.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Review mode changes nothing in the reviewed branch but still applies
mutants in its own clone, and a failing strengthened or gap test goes to
whoever assigned the run, or to the author in review mode. Deletions
wait until some mutant of the exercised code is caught, because zero
unique kills over nothing caught measures nothing however many mutants
survived. Survivor classes are defined by what an independent
requirement says about the change, after scoping errors and tool
artifacts are ruled out, and challenger mutants are proposed for the
catalog separately because neither mode edits skills.
`code-review` routes C# test changes to `test-reviewer`, the fuzz rule
defers to the language reference where no harness exists, and the
Stryker notes confirm the clean run discovered tests. Restated
definitions and discoverable lines are dropped from the references.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A protected test the pruner cannot resolve would otherwise lose its
only marker and turn into an unmarked test every review flags, so it
keeps the marker and is listed as unresolved in the oracle table.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contracts may now be named `CONTRACT.<name>.md`, so a `contract`
marker accepts any `CONTRACT*.md` rather than only `CONTRACT.md`.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A scaffold test now gains a strong source, becomes `mutant` when it
alone catches a fault, or is deleted before its pull request merges.
One that checks a property or intention no document states prompts a
question about where to specify it, and is marked with that source once
it exists, because a confirmation is not a source. Protected tests take
that path instead of keeping their scaffold marker, which a merge check
could not tell apart from a forgotten one.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Markers sit in test files at every depth, so a path relative to the
test would read differently from each and could not be resolved without
knowing where the marker lives. Marker paths are now repository-root
relative with forward slashes and exact case, a `spec` outside `docs/`
is named rather than given as a path, a crash file sits under `fuzz/`,
and a section names a heading or the number it starts with.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A `regression` marker names a Jira key, a GitHub issue as `#123` or
`owner/repo#123`, or a crash file under `fuzz/`, the forms a marker
check can resolve without guessing.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A `property` marker now names one of six well-known laws, because a
named law is its own definition while any other claimed property needs a
written source, and stays `scaffold` until it has one. An assertion
whose value comes from running the code is never a property, even over
generated inputs, and a round-trip never replaces a codec's vectors.
A section reference also matches an `<a id>` anchor placed before a
heading, and markers cite the anchor when the file has one, since it
survives a renamed heading or a renumbered section.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A marker can now cite a file in another repository as
`<owner>/<repo>@<sha>:<path>`, pinned to a commit of at least 12 hex
digits rather than a branch or tag, because a pinned commit never
changes and resolves the same every time. When the pinned section
changes upstream, the test is re-checked against the new text and its
pin or the test changes, never the expected value alone, since a value
changed without its pin cites text that decides the old one.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The test skills stay one skill per job rather than gaining a copy per language, because the method is identical in Rust and C# and only the mechanics differ.
Each test, case, or assertion carries an
// oracle:marker naming the source of its expected value. Tests added to catch a mutant are marked weak, and any intentional change may delete them without justification, because coverage driven by mutants rewards tests that pin how the code works today; teststest-prunerprotects stay protected whatever their marker. Tests written only for feedback while implementing never merge: they gain a source or are deleted first.Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com