Repositories are graded down for their own security test fixtures

[velzepooz]

What happens

A repository is graded down for the security fixtures it ships to test a scanner.

Scanning github.com/skilltrust/scan-action on skilltrust.app returns Security D, Perm hygiene D, and both findings come from the same place:

rule	severity	path
SD-017	high	tests/fixtures/malicious-repo/.claude/settings.json:1 — broad shell permission granted: Bash(curl *)
SD-007	high	tests/fixtures/malicious-repo/.claude/settings.json:3 — outbound network call detected

Both are correct readings of the file. The file exists precisely so that a scanner finds those things in it. The directory is called malicious-repo.

Why this is the engine's problem, not the scanned repo's

ADR-0002 already establishes that path gating is the mechanism for this exact class of error, and its Context section names the symptom:

findings fired on vendored READMEs, changelog examples, tutorial snippets and documentation that merely described a dangerous command. The signal was real but drowned. A scanner whose output users learn to skim is worth less than no scanner.

A security fixture is the same shape: content that deliberately exhibits the pattern, in a location that says so. The reasoning in ADR-0002 covers it; excludedDirs in pkg/rules/fileclass.go just does not list it. Today that list is node_modules/, .git/, vendor/, dist/, build/, .next/, target/ — build output and dependencies, no test-fixture term at all.

Blast radius

This is not one repository's cosmetic problem. Every project that tests security tooling is penalised for doing so: linters, scanners, policy engines, teaching repositories, and any project with a "here is what a bad config looks like" fixture. The public directory ranks and displays those grades, so the effect is systematic rather than incidental — and it lands hardest on exactly the repositories that take this seriously enough to write fixtures.

It also reads badly in the specific case above: our own Action's repository is publicly graded D by our own engine, for containing our own test data.

What needs deciding before anyone writes code

1. Which paths count as fixtures. testdata/ is unambiguous — it is a Go toolchain convention and the compiler already treats it as inert. Beyond that it is heuristics: tests/fixtures/, spec/fixtures/, __fixtures__/, examples/, */fixtures/*. Each one added is a directory name an attacker can use to park a real payload where the scanner will not look. The narrow, defensible start is testdata/ plus a path segment of exactly fixtures/; anything wider deserves its own argument.

2. Suppress or reclassify. Two different products:

• Gate it out — the finding never exists, the axis grade is unaffected, the report is quiet. Matches how excludedDirs behaves today.
• Keep it, mark it — the finding is reported as informational and excluded from the grade, so a reader can still see what is in there. Costs a wire-format change (ADR-0001 makes the four axes wire-stable) but does not blind anyone.

The second is more honest and more expensive. Worth an explicit choice rather than defaulting to the first because it is easier.

3. What happens to grades already issued. The corpus and every /r/{owner}/{repo} page hold grades computed under the current rule. Recomputing changes published numbers; not recomputing leaves the directory disagreeing with the engine. Either is defensible, neither is free.

Notes

• Affects the two rules in pkg/rules/exfiltration.go (SD-007) and pkg/rules/settings_json.go (SD-017) at minimum, but the gate belongs in pkg/rules/fileclass.go so every rule inherits it — ADR-0002 is explicit that rules must not hand-roll path checks.
• pkg/ is a published API consumed by skilltrust and scan-action; a change here is downstream-visible and the engine version is pinned in three places.
• Reported from production after a real scan; filed here rather than in skilltrust because the grading behaviour is the engine's.