@e2e exclusion integrity: 23 whole-spec markers retire 30% of all scenarios, several on reasons that do not hold

[rubenvdlinde]

Summary

gate-19's diff-scoped figure for the open development → beta PR is 137.
That number is a small slice of softwarecatalog's real e2e debt, because a
large share of the spec corpus is retired by a handful of very wide
@e2e exclude markers — several of which rest on reasons that are not true.

Classifying every @e2e exclude in openspec/specs/*/spec.md by the nearest
preceding heading:

scope	markers	blast radius
whole-spec	23	212 of 696 scenarios (30%)
requirement-level	28	every sibling scenario of that requirement
scenario-level	77	one scenario each

Two known gate defects amplify this: ConductionNL/.github#345 (an
@e2e exclude is scored as positive coverage, indistinguishable from a
real test) and #356 (a requirement-level marker silently exempts every
sibling). So one sentence, written once, can mark dozens of scenarios "covered".

Reasons that do not hold

multi-org-membership — 13 scenarios

No live deployment in this change's implementation pass (the shared dev
Nextcloud instance was ...)

This describes the state of the world on one afternoon, not a property of
the subject under test. It was arguably true when written and cannot be true
indefinitely; nothing about these scenarios is unobservable.

fe-object-modals — 10 scenarios

Vue object-modal components (view/create/edit/merge/migrate/upload/download/
delete/lock/mass-op) ...

These are excluded from browser testing. A Vue modal is the single most
browser-observable artifact the app has — open it, fill it, submit it, assert
the DOM. This PR's tests/e2e/spec-coverage/catalog-ratings.spec.ts does
exactly that for SubmitReviewModal.vue, which sits in the same family.

fe-settings-ui and fe-shell-navigation — 8 requirement-level markers

... is an interaction ...

Excluding interactions from the interaction-testing gate is inverted. This is
the reasoning that gate-19 exists to prevent.

method-decomposition — 52 scenarios (the single largest)

PHP internal refactor (controller/service method decomposition into helpers)
— pure backend structural ...

Plausible on its face — but one marker retires 52 scenarios, and nobody has to
look at them individually again.

Why this was not fixed in the gate-19 PR

Removing 23 whole-spec markers inside a PR whose stated purpose is lowering the
gate-19 count would raise the number by hundreds and is indistinguishable from
either gaming or vandalism, depending on which way it moved. Each marker needs
its own judgement, and several will legitimately survive.

Suggested approach

1. Convert every whole-spec and requirement-level marker to
   scenario-level markers first. This changes no verdict but makes the blast
   radius visible and forces each scenario to be looked at once.
2. Re-examine the four groups above; the fe-* ones are prime candidates for
   real Playwright coverage.
3. Adopt the rule that held up well here: a reason naming a test artifact
   holds; a reason naming a state of the world rots. multi-org-membership
   is the clearest instance.

Method (reproducible)

Classify by nearest preceding heading (#### Scenario: / ### Requirement: /
pre-first-requirement = whole-spec) and count scenarios per spec file. Note
that the gate's own JSON field is exclude_reason, not reason — querying
the wrong key returns nothing and manufactures a dramatic false finding.

Related

Found while closing gate-19 for softwarecatalog (137 → 112). See also #484
(sixteen file-level @e2e tags claiming coverage a file says it does not
provide).