builder-host.spec.ts:44 is flaky — fails ~3 in 10 runs on development AND on PRs, and is routinely mistaken for a real E2E regression

[rubenvdlinde]

Summary

tests/e2e/builder-host.spec.ts:44 — "BuilderHost — hello-world journey ›
navigates to a hello-message detail page" — fails intermittently on both
development and pull-request runs, at roughly 3 in 10 runs. It is a flaky
test, not a regression, and not caused by any particular diff.

It is currently the reason PRs in this repo show a red E2E Tests (Playwright)
cell that no diff explains, which makes it expensive: it is indistinguishable
from a real failure until someone walks back several runs.

The failure

Error: detail page must render the seeded body text

expect(locator).toBeVisible() failed
Locator: getByText(/rendered by your first virtual app/i)
Expected: visible
Timeout: 10000ms
Error: element(s) not found

  65 |    page.getByText(/rendered by your first virtual app/i),
  66 |    'detail page must render the seeded body text',
> 67 |  ).toBeVisible({ timeout: 10_000 })

The sibling tests either side of it — :24 loads /builder/hello-world and
renders the seeded index page and :70 navigates to the form page from the
manifest menu — pass in every run observed, including the runs where :44
fails. So the app, the seed and the builder host are all up; it is the detail-page
navigation specifically that is racy.

Evidence

Every arm below executed the full suite. Log sizes are given because a run that
dies before the tests produces a ~58 KB log that greps identically to a clean one
— run 31538644392 is exactly that shape and is excluded as invalid, not
counted as a pass or a failure.

run	event / branch	log	tally	builder-host:44
31543299215	push · development	273 KB	181 / 2	✓
31524590763	push · development (#182 merge)	277 KB	180 / 3	✓
31520347084	push · development	272 KB	181 / 2	✓
31438032584	push · development	285 KB	179 / 4	✘
31431360507	push · development	271 KB	181 / 2	✓
31393789354	dispatch · development	284 KB	179 / 4	✓
31477003232	pull_request · #175	288 KB	186 / 4	✘
31548218547	pull_request · #176	274 KB	181 / 2	✓
31547464168	pull_request · #177	260 KB	182 / 1	✘
31538644392	push · development	58 KB	—	invalid — died at Checkout additional apps, zero tests executed

3 failures in 9 valid runs, on both scopes. Run 31438032584 is the load-bearing
one: it is a plain development push with no pull request involved, and :44
failed there. That rules out any explanation that requires a PR — including the
"moving additional-apps ref on PR runs" hypothesis, which should not be quoted
for this test.

What this is not

• Not caused by any diff. It has now failed on three trees with disjoint
  changes and passed on three others, including passing on feat(openbuild): English vocabulary across schema, code and tests #176 whose diff was
  the largest of them.
• Not the same failure as REQ-AUTD-008 is unimplementable as shipped: automation CRUD is admin-only at the OR layer, so no editor or owner can author or enable one #173. That one is automations-rbac.spec.ts, two
  tests, deterministic, every run — and is addressed by fix(rbac): openbuild authorised the automation write, OpenRegister refused it, the user got a 500 (#173) #177. The two are
  routinely confused because they surface in the same red cell.

Suggested direction

The locator waits 10 s for text rendered after a client-side navigation into the
builder host, which is a separate webpack entry (src/builder.js) and
therefore a cold bundle fetch rather than an SPA route change — the same property
version-lifecycle-ui.spec.ts documents in its own comments and budgets 30 s for.
A plausible fix is to wait on the navigation/bundle completing rather than on the
text alone.

⚠️ Please do not "fix" this by widening the timeout blindly or by retrying the
test. If the detail page genuinely renders late for real users, the timeout is
reporting a product problem and hiding it would be the wrong trade. Establish
which it is first — the cheap check is whether the body text is late or absent
when it fails.

Filed while sweeping the open PRs so that a red E2E cell in this repo can be
attributed instead of re-investigated each time.