Skip to content

test(e2e): gate-19 186 → 161 — 25 verified anchors, 1 refusal, 2 false anchors removed - #179

Merged
rubenvdlinde merged 3 commits into
developmentfrom
test/gate19-anchors-verified
Aug 11, 2026
Merged

test(e2e): gate-19 186 → 161 — 25 verified anchors, 1 refusal, 2 false anchors removed#179
rubenvdlinde merged 3 commits into
developmentfrom
test/gate19-anchors-verified

Conversation

@rubenvdlinde

Copy link
Copy Markdown
Contributor

gate-19: 186 → 161

Measured full scope (hydra-gates --full) with hydra-gates @ main (b8c7ead, 1735-line checker — the ref CI pins), against openbuild@development:

before  {scenarios: 767, covered: 72, excluded: 509, uncovered: 186}
after   {scenarios: 767, covered: 97, excluded: 509, uncovered: 161}

The 186 baseline reproduces run 31459739568's figure exactly.

⚠️ This PR's own gate-19 cell proves nothing about that. Gate-19 is diff-scoped; this PR changes no spec file, so its scope is empty and it will report PASS trivially. The evidence is the negative control below, which is scope-independent.

Negative control

state uncovered
current 161
tests/e2e/builder-undo-redo.spec.ts removed 172 (+11 — exactly its 11 anchors)
restored 161

Positive control

Corrupting one slug to #adding-a-step-groups-fields-by-BOGUS dropped covered 99 → 98; restoring returned it to 99. The gate validates slugs, so the other 24 are real matches — not tags it accepted without looking.

What the 25 anchors are

Not new tests. builder-undo-redo.spec.ts and form-editor-logic.spec.ts already drive these scenarios and already run green; they carried no @e2e reference, so the gate could not see them. Every mapping was re-derived by reading the test body against the scenario text — a title that resembles a scenario is not coverage of it.

  • builder-undo-redo — 11 anchors across 6 tests
  • form-editor-logic — 14 anchors across 5 tests

One of the 14 needed the test extended before it could be anchored honestly: deleting-a-step-returns-its-fields-to-the-unassigned-pool asserted the removal and the pool re-appearance but not the spec's final AND (save auto-assigns an unassigned field to the last step, with a warning mark). The surface exists — assignUnassignedFieldsToFinalStep() in src/services/manifestValidation/formLogic.js, called from PageDesignerHost.vue's save() — so the test now re-adds a step, asserts the pool note, saves, and polls the persisted manifest for the assignment.

One scenario refused, with no exclusion written

form-editor-logic::raw-json-authored-logic-survives-unrelated-editor-edits asserts a Design ↔ Raw JSON tab round-trip. That tab does not exist on /builder/:slug/pages; the raw editor is a sidebar tab on the VirtualAppDetail page, writing the Application object while the designer saves onto the ApplicationVersion. Different surface, different storage target.

Anchoring it would have been false. Excluding it would have claimed "a browser cannot observe this", which is not the reason — the reason is that the spec describes a surface the product does not have. So it stays uncovered, with an in-file comment explaining why, so nobody later "fixes" it with a false anchor.

Two anchors REMOVED — this costs the number and is the right call

save-as-template.spec.ts credited viewer-cannot-save-a-template and seeded-cards-remain-read-only to a test whose own comment, three lines below the tags, says it proves neither: the assertion is toHaveCount(0) on a button, under an admin session, on a page where the button is not scoped to appear. Filed as #178.

Removing them did not raise uncovered — 161 either way. Both scenarios are also swallowed by a requirement-level @e2e exclude whose stated reason is about something else (REQ-SAT-001's reason is about "the captured record's internals"; the scenario it exempts is about whether a viewer sees a button). The false anchor was a second green layer over a scenario that was already invisibly exempt.

Chasing that produced a fleet-wide finding, filed as ConductionNL/.github#356: 471 of openbuild's 509 exclusions (92.5%) share a reason with a sibling — 42 reasons, one covering 32 scenarios, several beginning "mixed spec — …". Also worth knowing when reading any gate-19 percentage: coverage_pct is covered / (scenarios − excluded), so openbuild's headline 27.9% is 9.4% (72/767) unqualified.

Not done here

No test was skipped, no timeout widened, no mode: 'serial', no waitForLoadState('networkidle'), and no exclusion added.

ESLint on all three changed files reports only the pre-existing n/no-unpublished-import on the @playwright/test import, which every test file in the repo carries.

The red E2E Tests (Playwright) job is a separate, pre-existing failure — diagnosed and fixed in #177, which is open for review.

… 2 that did not

gate-19 (@e2e traceability), FULL scope, measured with hydra-gates @ main
(b8c7ead) against openbuild@development:

  before  {scenarios: 767, covered: 72, excluded: 509, uncovered: 186}
  after   {scenarios: 767, covered: 97, excluded: 509, uncovered: 161}

186 -> 161. No test was skipped, no timeout widened, no exclusion added.

WHAT THE 25 ARE. Not new tests: `builder-undo-redo.spec.ts` and
`form-editor-logic.spec.ts` already drive these scenarios and already run green;
they carried no `@e2e` reference, so the gate could not see them. Every mapping
was re-derived by reading the test BODY against the scenario text, not by
matching titles.

  builder-undo-redo   11 anchors across 6 tests
  form-editor-logic   14 anchors across 5 tests

One of the 14 needed the test EXTENDED before it could be anchored honestly:
`deleting-a-step-returns-its-fields-to-the-unassigned-pool` asserted the removal
and the pool re-appearance but not the spec's final AND (save auto-assigns an
unassigned field to the last step). The surface exists
(`assignUnassignedFieldsToFinalStep()`, called from PageDesignerHost's save()),
so the test now re-adds a step, asserts the pool note, saves, and polls the
PERSISTED manifest for the assignment.

ONE SCENARIO WAS REFUSED, deliberately, with no exclusion written:
`form-editor-logic::raw-json-authored-logic-survives-unrelated-editor-edits`
asserts a Design <-> Raw JSON tab round-trip. That tab does not exist on
/builder/:slug/pages — the raw editor is a sidebar tab on a different page,
writing a different object. Anchoring it would have been false and excluding it
would have been a false statement about observability, so it stays uncovered and
an issue is the right next step.

AND TWO ANCHORS WERE REMOVED, which RAISES the covered count's honesty at the
cost of the number. `save-as-template.spec.ts` credited
`viewer-cannot-save-a-template` and `seeded-cards-remain-read-only` to a test
whose own comment, three lines below the tags, states that it proves neither:
the assertion is toHaveCount(0) on a button, under an ADMIN session, on a page
where the button is not scoped to appear. Filed as #178.

Removing them did NOT raise `uncovered` (161 either way) — because both
scenarios are ALSO swallowed by a requirement-level `@e2e exclude` whose stated
reason is about something else entirely. Measuring that led to the wider
finding: 471 of openbuild's 509 exclusions (92.5%) share a reason with a
sibling, 42 reasons covering them, one covering 32 scenarios. Filed upstream as
ConductionNL/.github#356.

NEGATIVE CONTROL (scope-independent):
  161  with tests/e2e/builder-undo-redo.spec.ts present
  172  with it removed          (+11 = exactly its 11 anchors)
  161  restored

Positive control that the gate validates slugs rather than the tag's presence:
corrupting one slug to #adding-a-step-groups-fields-by-BOGUS dropped covered
99 -> 98; restoring returned it to 99.

Refs #178
…dy proves

161 -> 160.

`component-blocks.spec.ts` anchored
`component-blocks::blocks-filter-shows-blocks-without-the-clone-action`. That
scenario lives in `openbuild-template-catalogue`, not `component-blocks`, so the
ref resolved to NOTHING: gate-19 credited no scenario and reported no error,
while the real scenario sat in the uncovered list and a green test that proves
it was sitting right there.

A dangling anchor is silent in both directions — it looks like coverage in the
file and like an absence in the gate. Found by walking every anchor in the suite
against the gate's own parser rather than by eye.

openbuild has 15 of them. This is the only one that was a simple mis-spelled
target. The other 14 are in `spec-coverage/page-designer-ui.spec.ts`, name slugs
that exist in no spec, sit on seven tests CI never runs (`test.skip(!LIVE)`,
`OPENBUILD_E2E_LIVE` is never set), and belong to ten scenarios that are ALL
`@e2e exclude`d on the grounds that another spec's Playwright tests cover them.
Repointing those while the tests stay skipped would move them from silently
uncounted to counted against tests that never execute, so they are filed rather
than patched — #181.

What this anchor does NOT claim is recorded next to it: the scenario's THEN lists
"name, description, category and a preview"; the test asserts the name and the
AND (no clone action). The distinguishing behaviour — browse-only, no clone
affordance — is covered.

Refs #181
@github-actions

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/openbuild @ dd18303

Check PHP Vue Security License Tests
lint
phpcs
phpmd
psalm
phpstan
phpmetrics
eslint
stylelint
build
check-manifest
test-l10n
check-gitignore
check-nc-floor
composer ✅ 101/101
npm ✅ 654/654
PHPUnit
Newman ⏭️
Playwright
Hydra gates

Quality workflow — 2026-08-11 13:48 UTC

Download the full PDF report from the workflow artifacts.

The new REQ-OBFEL-001 poll carried `timeout: 30_000`, which is exactly the
per-test budget in tests/e2e/playwright.config.ts. The test would always die
first, so the poll's own message — "saving must append the still-unassigned
field keys to the final step" — could never be printed, and the failure would
surface as a bare test timeout instead.

That is the precise failure mode the config's shorter `expect` timeout exists to
prevent; its own comment says so. Use the repo default, 15_000.

Not a widening: this LOWERS a number, and it makes the assertion able to report
itself.
@github-actions

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/openbuild @ 40d4357

Check PHP Vue Security License Tests
lint
phpcs
phpmd
psalm
phpstan
phpmetrics
eslint
stylelint
build
check-manifest
test-l10n
check-gitignore
check-nc-floor
composer ✅ 101/101
npm ✅ 654/654
PHPUnit
Newman ⏭️
Playwright
Hydra gates

Quality workflow — 2026-08-11 13:51 UTC

Download the full PDF report from the workflow artifacts.

@github-actions

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/openbuild @ 6f6f6d2

Check PHP Vue Security License Tests
lint
phpcs
phpmd
psalm
phpstan
phpmetrics
eslint
stylelint
build
check-manifest
test-l10n
check-gitignore
check-nc-floor
composer ✅ 101/101
npm ✅ 654/654
PHPUnit
Newman ⏭️
Playwright
Hydra gates

Quality workflow — 2026-08-11 14:26 UTC

Download the full PDF report from the workflow artifacts.

@rubenvdlinde
rubenvdlinde merged commit ff25c1b into development Aug 11, 2026
31 of 33 checks passed
@rubenvdlinde
rubenvdlinde deleted the test/gate19-anchors-verified branch August 11, 2026 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant