Skip to content

STU-372: Add deterministic browser fixtures for the redesigned user journeys - #76

Closed
sgardoll wants to merge 15 commits into
mainfrom
sgardoll/stu-372-add-deterministic-browser-fixtures-for-the-redesigned-user
Closed

sgardoll wants to merge 15 commits into
mainfrom
sgardoll/stu-372-add-deterministic-browser-fixtures-for-the-redesigned-user

Conversation

@sgardoll

@sgardoll sgardoll commented Sep 22, 2026 •

Copy link
Copy Markdown
Owner

Closes STU-372.

What this PR does

  • Adds Playwright configuration (playwright.config.js) that runs against a local Vite dev server.
  • Adds e2e/fixtures/apiFixtures.js with deterministic, contract-shaped mocks for auth, subscription, generation, review, and deploy endpoints.
  • Adds e2e/smoke.spec.js baseline tests for the redesigned hero landing and composer state.
  • Adds test:ui / test:ui:headed scripts and installs @playwright/test as a dev dependency.
  • Fixtures never invoke paid generation, billing, or real project writes.

Verification

  • npm test — 274/274 passing.
  • npm run build — production bundle built successfully.
  • npm run test:ui — 6/6 passing.
  • A deliberately broken assertion was shown to fail and then restored, proving the tests exercise real state.

Notes

The single browser warning (Permissions policy violation: compute-pressure) is filtered as an unrelated Chromium notice.

RetriggerConfidence Score: 5/5

The PR appears safe to merge; no new actionable defect was introduced by the changes since the previous review.

Fix All in DevinFindings

  1. P2 Exact Checks Look Approximate ▶
Fix with agent prompt
### Issue 1
src/customCodeVerification.js:217-220
These notices are generated from every unrepresentable project dependency without considering the classes being deployed. A deployment containing only `dart:`-importing classes is therefore logged and presented as “Verified approximately” even though its verification is exact. This false warning makes the deployment diagnostics less reliable. Only emit an approximation notice when at least one verified source uses package resolution.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

This PR adds deterministic Playwright fixtures and baseline browser coverage for the redesigned user journeys.

  • Runs browser tests against a local Vite server and includes them in CI.
  • Provides contract-shaped fixtures for authentication, subscriptions, generation, review, projects, and deployment.
  • Blocks unrecognized external traffic and serves SRI-pinned dependencies from verified local copies.
  • Extends custom-code dependency parsing and verification diagnostics, with corresponding tests and deployment documentation.
  • Greptile automatically discovered a related ticket that helped explain the purpose of this PR: provide credential-free, deterministic browser fixtures that cannot invoke paid generation, billing, or real project writes.

Reviews (4) · Last reviewed commit: "test(e2e): serve SRI-pinned assets from ..."

STU-368. Which packages the Flutter SDK supplies was decided by a
hand-maintained name list kept in three places, and flutter_web_plugins was in
none of them. Every project declaring it - which is every web-enabled project
with a plugin - read it as a dependency that could not be reproduced, and the
deploy was refused for a package most of its classes never imported.

The project's own pubspec already says where each package comes from; the
parser read those lines and threw them away. It now returns them as
sourceKey/sourceValue, so the classification is exact and cannot fall behind
the SDK:

  sdk:                            -> the Flutter SDK, reproduced from the name
  a nested version:               -> a constraint written on its own line
  git: / path: / nested hosted:   -> not expressible in the manifest

SDK_DEPENDENCIES is gone. The runner's allowedSdkPackages is gone too - it
independently refused the same deploy with 'Unsupported SDK package'. That
allowlist was also the only validation of an sdkPackages entry, so the package
name pattern now does that job; without the swap a caller-supplied name would
reach the generated pubspec unchecked. The runner still writes 'sdk: flutter'
itself, so that value never comes from a caller.

Refusal is now scoped per class, to that class's own imports, so one git: entry
no longer leaves an entire deploy uncompiled.

The accept/warn split already existed and is unchanged: a real analyzer error
refuses with 422, and a class that could not be compiled is pushed with its
warning logged before the provisioning request.

Corrects DEPLOYMENT.md, which still described the scaffolding-closure
compilation that reverting a1238f0 removed, and still claimed that nothing
unverified is deployed.

Verified: 274/274 tests, deploy guard 9/9, proxies 30/30, build clean, dart
analyze on the runner reports no issues.
…- Add Playwright config with local Vite webServer\n- Add npm run test:ui headless browser command\n- Add reusable API fixtures: guest, Free/Pro/Power, unresolved,\n one/many artifacts, missing review score, rejected generation,\n partial/unknown deploy, project list/export/sync\n- Add baseline smoke tests for hero landing and composer state\n- Keep tests independent of future markup; no paid generation/billing/project writes\n\nSTU-372
…ions\n\nThe compute-pressure permissions-policy notice is a browser-level\nwarning unrelated to app code. Filter it from the global console\nfailure assertion so smoke tests stay green.\n\nSTU-372
@sgardoll

Copy link
Copy Markdown
Owner Author

@greptileai

Comment thread e2e/fixtures/apiFixtures.js Resolved
Comment thread src/pubspecSync.js Resolved
Comment thread e2e/fixtures/apiFixtures.js Resolved
Comment thread .github/workflows/ci.yml Resolved
…can reach a class transitively

(cherry picked from commit 9d338ad)
The default fixture router continued every unrecognized request, so an
endpoint missing from the fixture map (runpipeline-image, the staging
FlutterFlow sync, an env-overridden deploy URL) could silently reach a
paid generation or a real project. The catch-all now aborts unmatched
requests so un-fixture'd calls fail tests loudly.

Only three request classes are allowed through, all explicit:
- fixture-map URLs are fulfilled as before;
- static assets index.html references are fulfilled with inert stubs
  (Tailwind, Google Fonts CSS, YouTube embeds), removing the page
  load's dependency on live CDNs and killing the YouTube player/ad/
  tracking request cascade;
- same-origin requests the local Vite dev server serves (the document,
  modules, Vite internals, public assets) continue, except /api paths,
  which vite.config.js proxies to real AI providers;
- the SRI-pinned static scripts (JSZip, highlight.js, FingerprintJS)
  continue because a local stub can never match the pinned digest and
  app.js needs FingerprintJS to resolve identity; SRI verifies them
  byte-for-byte.

Verified by rerunning the smoke suite (6 passed, 10x faster with the
YouTube cascade gone) and by probing that fetches to runpipeline-image,
the staging sync and /api proxy paths all abort.

STU-372
buildArtifactBundle synthesized deployOrder values like "file-1" while
the artifacts themselves get real IDs such as
"custom-action-greet-user". Normalization preserves unknown order IDs,
and the deployment planner matches order entries against artifact.id,
filtering out every synthetic value — so a deploy journey driven by
these fixtures produced an empty deployment plan instead of exercising
artifact deployment.

Map the artifacts first and derive deployOrder from the mapped
artifacts' IDs, the same relationship src/artifactBundle.js guarantees.
Verified against the real contract: oneArtifact now plans
lib/custom_code/actions/GreetUser.dart and manyArtifacts plans all
three files in class -> action -> widget order with no planner errors.

STU-372
The pull-request workflow ran only unit, build, deploy-guard, proxy and
leak-scan checks, so regressions in fixture routes, selectors and smoke
assertions could merge while CI stayed green.

Add the Chromium browser install and npm run test:ui after the
production build, so the deterministic browser suite gates every pull
request. Step style and the pinned-action/least-permission hardening
from the earlier CI commits are unchanged.

STU-372
Comment thread src/customCodeVerification.js Resolved
Comment thread e2e/fixtures/apiFixtures.js Resolved
… scratch-graph approximation

(cherry picked from commit e300b76)
Comment on lines +217 to +220
const approximate = unrepresentable.map(
(name) =>
`Your project declares ${name} from a source that cannot be reproduced outside it, so the pre-deploy compile check runs against a scratch package graph that omits ${name}. Classes that do not import ${name} directly are still checked, but against that reduced graph, so their transitive package resolution may differ from your project's and the check is approximate for them.`,
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Exact Checks Look Approximate

These notices are generated from every unrepresentable project dependency without considering the classes being deployed. A deployment containing only dart:-importing classes is therefore logged and presented as “Verified approximately” even though its verification is exact. This false warning makes the deployment diagnostics less reliable. Only emit an approximation notice when at least one verified source uses package resolution.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/customCodeVerification.js
Line: 217-220

Comment:
**Exact Checks Look Approximate**

These notices are generated from every unrepresentable project dependency without considering the classes being deployed. A deployment containing only `dart:`-importing classes is therefore logged and presented as “Verified approximately” even though its verification is exact. This false warning makes the deployment diagnostics less reliable. Only emit an approximation notice when at least one verified source uses package resolution.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Claude Code Fix in Codex Fix in Devin

Greptile P1: the fixture router let the five SRI-pinned third-party assets
(including FingerprintJS, required during page initialization) continue to
the live network, so a blocked or rate-limited CDN could fail
otherwise-correct tests. Vendor the pinned bytes in e2e/fixtures/vendor/ —
each verified against index.html's integrity attribute — and fulfill them
locally. The same-origin Vite dev server is now the only request still
allowed through; everything else fulfills from a fixture, an inert stub,
vendored bytes, or aborts.

STU-372
@sgardoll

Copy link
Copy Markdown
Owner Author

@greptileai — all findings from the previous review have been addressed (see new commits). Please re-review this PR.

@sgardoll

Copy link
Copy Markdown
Owner Author

Closing: this branch is already fully contained in milestone/stu-implement-updated-ui-ux, merged during the earlier integration run as b15651b (STU-372). GitHub refuses to retarget this PR to the milestone branch for that exact reason — there are no new commits between the two. The work therefore ships with the milestone → main PR and this PR no longer corresponds to anything unmerged. Nothing is lost; the commits remain in history.

@sgardoll sgardoll closed this Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant