diff --git a/README.ko.md b/README.ko.md index bfb1727..25a19bc 100644 --- a/README.ko.md +++ b/README.ko.md @@ -91,13 +91,20 @@ $EDITOR .sdlc/config.md # build, test, lint, run, e2e, ## 설정 프로젝트마다 다른 값은 `.sdlc/config.md` 한 곳에만 둡니다. `build`, `test`, -`lint`, `run`, `e2e` 명령, 화면이나 API를 다룰 `qa` 도구, `human_gates`(기본값 +`lint`, `run`, `e2e` 명령, 화면이나 API를 다룰 `qa` 도구, 선택 실행기 어댑터 `qa_runner`와 +그 로그인 명령 `qa_session`, `human_gates`(기본값 `intent`, 모든 plan을 직접 승인하려면 `plan` 추가, 위험 플래그가 있는 plan은 설정과 상관없이 사람이 승인), `knowledge` 폴더, `repos`(리뷰가 묶을 저장소. 프로젝트 루트 아래 저장소가 많으면 나열하거나 리뷰마다 `sdlc.sh review pass --repos "<경로> ..." "<메모>"`로 지정하고, 비우면 루트 아래 전부)입니다. `.sdlc/`는 `.git/info/exclude`로 git에서 빠집니다. 팀이 배운 것을 공유하려면 `knowledge:`를 git이 추적하는 폴더로 지정하세요. +`adapters/e2e/`가 그런 실행기 하나입니다. `qa_runner: e2e`이면 QA는 브라우저·모바일 +시나리오마다 [e2e](https://www.npmjs.com/package/e2e) 테스트를 씁니다. 결정적 층(증거, +모델 없음)과, 다음 라운드부터 모델 호출 없이 재생되는 선택 agent 층으로 나뉩니다. +키트는 e2e의 에이전트 가이드를 버전 고정으로 들고 있어(Apache-2.0) 오프라인에서도 +쓸 수 있고, `adapters/e2e/update.sh [버전]`으로 갱신합니다. + ## 늘 지키는 규칙 - **권한은 게이트와 별개입니다.** 데이터 유실, 마이그레이션, 공개 API 변경, diff --git a/README.md b/README.md index b90dcac..c898b35 100644 --- a/README.md +++ b/README.md @@ -93,6 +93,7 @@ Closed features move to `.sdlc/archive/`. Verified facts move to the `.sdlc/config.md` is the only project-specific file: the `build`, `test`, `lint`, `run` and `e2e` commands, the `qa` tool that drives screens or APIs, +an optional `qa_runner` adapter with its `qa_session` sign-in command, `human_gates` (default `intent`; add `plan` to approve every plan yourself — a plan with a risk flag always asks), the `knowledge` folder, and `repos` (the repositories a review binds — list them when the project root holds many, @@ -101,6 +102,13 @@ empty means every one under it). `.sdlc/` stays out of git through `.git/info/exclude`; point `knowledge:` at a tracked folder to share what the team learns. +`adapters/e2e/` is one such runner: with `qa_runner: e2e`, QA writes each +browser or mobile scenario as an [e2e](https://www.npmjs.com/package/e2e) +test with a deterministic layer (the evidence, no model) and an optional +agent layer that replays without model calls on later rounds. The kit +vendors e2e's agent guide (Apache-2.0, pinned) so it works offline; +`adapters/e2e/update.sh [version]` refreshes it. + ## Rules that never relax - **Authority is not a gate.** Data loss, migrations, public API changes, diff --git a/SKILL.md b/SKILL.md index 8d6f45d..1e41d65 100644 --- a/SKILL.md +++ b/SKILL.md @@ -75,7 +75,9 @@ finds `.git`. ## Config and records `.sdlc/config.md` holds every project value: `build`, `test`, `lint`, `run`, -`e2e`, `qa` (the tool that drives screens or APIs), `human_gates` (default +`e2e`, `qa` (the tool that drives screens or APIs), `qa_runner` (an optional +adapter in `adapters/`, such as `e2e`) and `qa_session` (its sign-in command), +`human_gates` (default `intent`), `knowledge` (the durable knowledge folder) and `repos` (the repositories a review binds; empty means every one under the project root; `review --repos` names them for one review). diff --git a/VERSION b/VERSION index 197c4d5..437459c 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.4.0 +2.5.0 diff --git a/adapters/e2e/README.md b/adapters/e2e/README.md new file mode 100644 index 0000000..d22cc66 --- /dev/null +++ b/adapters/e2e/README.md @@ -0,0 +1,74 @@ +# Adapter: e2e (optional QA runner) + +Use when config.md says `qa_runner: e2e`. QA then writes each browser or +mobile scenario as an [e2e](https://www.npmjs.com/package/e2e) test, so a +later round re-runs it with one command instead of re-driving it by hand. +Without that key, nothing here applies. + +The kit carries what an agent needs to write and run these tests offline: +`vendor/e2e-skill/` is e2e's own agent guide (Apache-2.0, `LICENSE` and +`NOTICE` inside), pinned in `VERSION`. `update.sh [version]` refreshes it from +npm. The project still installs the runner itself. + +## Set up once per project + +1. Node `^22.22.3 || >=24.8.0` (see `VERSION`). Install the pinned packages + as dev dependencies: `e2e`, `@e2e-dev/web` (or `@e2e-dev/mobile`). +2. Turn telemetry off for every run: `E2E_TELEMETRY_DISABLED=1`. +3. Copy `templates/e2e.config.ts` and fill the app URL. Agent steps need a + model: prefer a subscription login the human makes once + (`npx e2e login openai`, or another plan listed in + `vendor/e2e-skill/references/setup.md`). Deterministic tests need none. +4. Sign-in the runner cannot drive (a popup, SSO, a device prompt) goes in + `qa_session:` — a command that writes the signed-in storage to a file the + setup test injects (`templates/session.setup.e2e.ts`). e2e does not follow + windows the app opens. + +Read `vendor/e2e-skill/SKILL.md` and the one reference it points to before +writing tests. + +## Popups + +e2e does not follow windows the app opens. `templates/popup-bridge.ts` is an +init script that turns `window.open` into a full-viewport iframe in the same +tab and makes `window.opener`, `close()` and `focus()` behave inside it, so a +same-origin popup that talks through opener and postMessage works unchanged; +assert inside it with `browser.frameLocator`. Its header lists what it does +not cover (cross-origin popups, `target="_blank"` links, `noopener`) and how +it differs from a real popup. Tried on two document-viewer popups: the +deterministic run passed twice, and an agent step inside the popup replayed +with no model call on its second run. + +## Each scenario, two layers + +`templates/scenario.e2e.ts` shows both, tagged with the scenario ID: + +- **det** — exact steps and locator, URL and request assertions; no model. + This is the evidence: its result is the scenario's PASS or FAIL. +- **agent** (optional) — `agent.act` for steps whose wording or layout + churns, each followed by a locator assertion so the replay cache records + it. The first run calls the model; later runs replay with no model call + until the screen changes. `agent.assert` always calls the model. + +Run det in every round; run agent with `--strict-cache` when it must not +spend model calls (a stale recording then fails instead of re-planning). +Save `--reporter json` output in `logs/` with the scenario ID. + +## Limits seen in use + +- Opening is not always read-only. In the trial, merely opening a one-page + assignment in its viewer marked it submitted on the server. Before driving + a flow on shared or seeded data, find out what opening, closing and + navigating write, and use fixture accounts whose state may change. +- An agent step absorbs a renamed control and still passes; only the det + locator catches that regression. Keep det as the gate. +- A wrong expectation after a correct replay evicts the correct recording; + the next run pays the model again. Fix expectations before re-running. +- A replay that cannot find its target waits for it before handing over and + is counted as missed, not handed off. +- Sessions keep cookies and localStorage, not sessionStorage; inject that in + the test when the app needs it. Signed-in tokens expire: re-run + `qa_session` when the setup assertion fails. +- The replay cache stores typed inputs in plain text. Commit it only as + reviewed test data, or keep it local. +- e2e is pre-1.0: pin versions; update with `update.sh` and re-run the suite. diff --git a/adapters/e2e/VERSION b/adapters/e2e/VERSION new file mode 100644 index 0000000..98be5d0 --- /dev/null +++ b/adapters/e2e/VERSION @@ -0,0 +1,5 @@ +e2e: 0.20.0 +web: 0.14.0 +node: ^22.22.3 || >=24.8.0 +source: https://www.npmjs.com/package/e2e (skills/e2e, LICENSE, NOTICE) +updated: 2026-10-14 diff --git a/adapters/e2e/templates/e2e.config.ts b/adapters/e2e/templates/e2e.config.ts new file mode 100644 index 0000000..f0f0ee1 --- /dev/null +++ b/adapters/e2e/templates/e2e.config.ts @@ -0,0 +1,24 @@ +// sdlc-kit adapter template — fill APP_URL and, for agent steps, a model. +import type { E2EConfig } from 'e2e'; +import { web } from '@e2e-dev/web'; +// Agent steps only. A subscription login made once by the human, e.g.: +// npx e2e login openai then import { chatgpt } from 'e2e/oauth/chatgpt'; +// import { chatgpt } from 'e2e/oauth/chatgpt'; + +const APP_URL = process.env.APP_URL ?? 'http://localhost:3000'; + +export default { + targets: [ + { + name: 'web', + engine: web(), + // A fixed identity keeps replay recordings valid when the host changes (dev, staging). + app: { url: APP_URL, identity: 'my-app' }, + }, + ], + // agents: { default: { model: chatgpt(''), context: 'Read-only QA: never create, submit or delete data.' } }, + workers: 1, // one signed-in account shared by every test + retries: 0, + cache: { mode: 'read-write', dir: '.e2e/cache' }, + reporters: ['list'], +} satisfies E2EConfig; diff --git a/adapters/e2e/templates/popup-bridge.ts b/adapters/e2e/templates/popup-bridge.ts new file mode 100644 index 0000000..51201c3 --- /dev/null +++ b/adapters/e2e/templates/popup-bridge.ts @@ -0,0 +1,147 @@ +// sdlc-kit adapter template — popup bridge. +// e2e does not follow windows the app opens. This init script turns window.open into a +// full-viewport iframe in the same tab, with no app change, so popup flows stay testable. +// +// Top window: +// - window.open(url, name, features) makes (or reuses, by name) a full-viewport iframe +// and returns its contentWindow; '_self' | '_parent' | '_top' go to the native open; +// '' | '_blank' always make a new iframe. +// Inside a bridge iframe: +// - window.opener is the window that opened it; window.close() removes the iframe +// (the opener's reference then reports closed === true); window.focus() raises it; +// window.open() opens a nested bridge popup. +// Other iframes the app uses are left alone. +// +// Works for same-origin popups that talk through opener, postMessage, window.name and +// .closed. Not covered: cross-origin popups (frameElement is unreadable, and the server +// may forbid framing), and form targets (no window.open), noopener. +// Differences from a real popup: sessionStorage is shared, not copied; size features are +// ignored; window.top is the app window; the app window gets no blur/visibilitychange; +// the popup covers the app until it closes. +// +// Use: +// await installPopupBridge(browser); // before app.open +// browser.frameLocator(popupFrameSelector('name')) // assert inside the popup +// await browser.evaluate(listPopups) // open popups +// toBeVisible passes on an element with opacity 0: assert something that proves the +// popup painted (an image loaded, computed opacity) before trusting a screenshot. + +/** Self-contained: browser.addInitScript serializes it into every document, frames included. No outside references. */ +export function popupBridgeInit(): void { + const W = window as unknown as Record; + if (W.__popupBridgeInstalled) return; + W.__popupBridgeInstalled = true; + const ATTR = 'data-popup-bridge'; + + if (window === window.top) { + const nativeOpen = window.open.bind(window); + const frames = new Map(); + const openers = new Map(); + let seq = 0; + let z = 2147480000; + const frameOf = (win: unknown) => { + for (const f of frames.values()) if (f.contentWindow === win) return f; + return null; + }; + const raise = (f: HTMLIFrameElement) => { + f.style.zIndex = String(++z); + }; + const api = { + open(url: string, name: string | undefined, features: string | undefined, opener: Window): Window | null { + const target = name ?? ''; + if (target === '_self' || target === '_parent' || target === '_top') return nativeOpen(url, target, features); + const anonymous = target === '' || target === '_blank'; + const key = anonymous ? `__anonymous_${++seq}` : target; + const href = url ? new URL(url, location.href).href : ''; + let f = frames.get(key); + if (f && f.isConnected) { + if (href) f.src = href; + } else { + f = document.createElement('iframe'); + if (!anonymous) f.name = target; + f.setAttribute(ATTR, key); + f.setAttribute('title', `popup ${key}`); + if (features) f.setAttribute('data-popup-features', features); + f.style.cssText = 'position:fixed;inset:0;width:100vw;height:100vh;border:0;margin:0;padding:0;background:#fff;'; + f.src = href || 'about:blank'; + (document.body ?? document.documentElement).appendChild(f); + frames.set(key, f); + } + openers.set(f, opener); + raise(f); + return f.contentWindow; + }, + close(win: Window): void { + for (const [k, f] of frames) { + if (f.contentWindow === win) { + frames.delete(k); + openers.delete(f); + f.remove(); + } + } + }, + focus(win: Window): void { + const f = frameOf(win); + if (f) raise(f); + }, + openerOf(win: Window): Window | null { + const f = frameOf(win); + return f ? (openers.get(f) ?? window) : null; + }, + list(): { key: string; name: string; src: string; url: string; z: string }[] { + return [...frames].map(([key, f]) => { + let url = ''; + try { + url = f.contentWindow?.location.href ?? ''; + } catch { + url = '(cross-origin)'; + } + return { key, name: f.name, src: f.src, url, z: f.style.zIndex }; + }); + }, + }; + W.__popupBridge = api; + window.open = function (url?: string | URL, name?: string, features?: string) { + return api.open(url == null ? '' : String(url), name, features, window); + } as typeof window.open; + return; + } + + // Child document: act only inside an iframe the bridge made (frameElement is readable only same-origin). + let host: Element | null = null; + try { + host = window.frameElement; + } catch { + host = null; + } + if (!host || !host.hasAttribute(ATTR)) return; + const api = (window.top as unknown as Record).__popupBridge; + if (!api) return; + Object.defineProperty(window, 'opener', { configurable: true, get: () => api.openerOf(window), set: () => undefined }); + window.close = () => api.close(window); + window.focus = () => api.focus(window); + window.open = function (url?: string | URL, name?: string, features?: string) { + return api.open(url == null ? '' : String(url), name, features, window); + } as typeof window.open; +} + +type InitScriptHost = { addInitScript(script: () => unknown): Promise }; + +/** Call before app.open; every later document gets the bridge. */ +export async function installPopupBridge(browser: InitScriptHost): Promise { + await browser.addInitScript(popupBridgeInit); +} + +/** Selector for a bridge iframe by window name; without a name, the newest one. */ +export const popupFrameSelector = (name?: string): string => + name ? `iframe[data-popup-bridge="${name}"]` : 'iframe[data-popup-bridge]:last-of-type'; + +/** browser.evaluate(listPopups) lists the open bridge popups. */ +export const listPopups = (): { key: string; name: string; src: string; url: string; z: string }[] => + ((window as unknown as Record).__popupBridge?.list() ?? []) as { + key: string; + name: string; + src: string; + url: string; + z: string; + }[]; diff --git a/adapters/e2e/templates/scenario.e2e.ts b/adapters/e2e/templates/scenario.e2e.ts new file mode 100644 index 0000000..2deff6c --- /dev/null +++ b/adapters/e2e/templates/scenario.e2e.ts @@ -0,0 +1,29 @@ +// sdlc-kit adapter template — one PLAN.md scenario, two layers, tagged with its ID. +// det: npx e2e run --tag det --reporter json (evidence; no model) +// agent: npx e2e run --tag agent --strict-cache (optional; replays without model calls) +import { describe, test } from '@e2e-dev/web'; +import { expect } from 'e2e'; + +describe('S1 ', { session: 'user', tags: ['S1'] }, () => { + test('det', { tags: ['det'] }, async ({ app, browser }) => { + await app.open('/start'); + await browser.locator('nav >> text=Reports').tap(); + await expect(browser).toHaveURL('/reports'); + // A request the change must (or must not) send: record it, then assert. + const sent: string[] = []; + await browser.route('**/api/audit-log', async (route) => { + if (route.request.method === 'POST') sent.push(route.request.postData ?? ''); + await route.continue(); + }); + await browser.locator('text=Monthly').tap(); + await expect.poll(() => sent.length).toBeGreaterThan(0); + }); + + test('agent', { tags: ['agent'] }, async ({ app, agent, browser }) => { + await app.open('/start'); + // Keep the act wording stable: it keys the replay recording. + await agent.act('open the Reports page from the main menu'); + // A locator or URL check after the act is what records it for replay. + await expect(browser).toHaveURL('/reports'); + }); +}); diff --git a/adapters/e2e/templates/session.setup.e2e.ts b/adapters/e2e/templates/session.setup.e2e.ts new file mode 100644 index 0000000..3336790 --- /dev/null +++ b/adapters/e2e/templates/session.setup.e2e.ts @@ -0,0 +1,25 @@ +// sdlc-kit adapter template — restore a sign-in that config `qa_session:` captured +// (popup, SSO or device flows the runner cannot drive), then save it as a named session. +import { readFileSync } from 'node:fs'; +import { test } from '@e2e-dev/web'; +import { expect } from 'e2e'; + +// Written by the qa_session command: { "localStorage": {...}, "sessionStorage": {...} } +const STATE_FILE = process.env.QA_SESSION_FILE ?? '.auth/session.json'; + +test.setup('signed-in session', { sessions: ['user'] }, async ({ app, browser, session }) => { + const state = JSON.parse(readFileSync(STATE_FILE, 'utf8')) as { + localStorage?: Record; + sessionStorage?: Record; + }; + await app.open('/'); + await browser.evaluate((s: typeof state) => { + for (const [k, v] of Object.entries(s.localStorage ?? {})) localStorage.setItem(k, v); + for (const [k, v] of Object.entries(s.sessionStorage ?? {})) sessionStorage.setItem(k, v); + return true; + }, state); + await app.open('/'); // reload signed in + // Assert a signed-in element before saving; session.save does not check it. + await expect(browser.locator('[data-signed-in], nav')).toBeVisible(); + await session.save('user'); +}); diff --git a/adapters/e2e/update.sh b/adapters/e2e/update.sh new file mode 100755 index 0000000..3d3975d --- /dev/null +++ b/adapters/e2e/update.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# Refresh the vendored e2e agent guide from the npm package, pinned by version. +# Usage: adapters/e2e/update.sh [e2e-version] (default: the version in adapters/e2e/VERSION) +# Needs npm and tar only while updating; using the vendored guide needs neither. +set -eu +DIR=$(cd "$(dirname "$0")" && pwd) +case "${1:-}" in -h|--help) sed -n '2,4p' "$0" | sed 's/^# //'; exit 0;; esac +current=$(awk -F': *' '$1=="e2e"{print $2}' "$DIR/VERSION") +version=${1:-$current} +[ -n "$version" ] || { echo 'update.sh: no e2e version given or pinned' >&2; exit 2; } +tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT +( cd "$tmp" && npm pack --silent "e2e@$version" >/dev/null ) +tar -xzf "$tmp"/e2e-*.tgz -C "$tmp" +src=$tmp/package +[ -f "$src/skills/e2e/SKILL.md" ] || { echo "update.sh: e2e@$version has no skills/e2e/SKILL.md" >&2; exit 1; } +dest=$DIR/vendor/e2e-skill +rm -rf "$dest.new"; mkdir -p "$dest.new" +cp -R "$src/skills/e2e/." "$dest.new/" +cp "$src/LICENSE" "$src/NOTICE" "$dest.new/" +rm -rf "$dest"; mv "$dest.new" "$dest" +# @e2e-dev/web declares a peer range on e2e; record the latest web release to install beside it. +web=$(npm view @e2e-dev/web version 2>/dev/null || true) +node=$(awk -F'"' '/"node"/{print $4; exit}' "$src/package.json") +cat > "$DIR/VERSION" <`. + +| When you are | Topic | Read | +| --- | --- | --- | +| Adding e2e to a project, writing `e2e.config.ts`, starting the app from the config, signing in to a model, setting up mobile targets | `setup` | [references/setup.md](references/setup.md) | +| Looking at the running app, checking a UI change, taking a screenshot or video, or trying a locator before writing it, over `e2e mcp` | `mcp` | [references/mcp.md](references/mcp.md) | +| Writing or fixing a test: fixtures, locators, actions, matchers, sign-in sessions, the `browser` fixture, API-only tests | `writing-tests` | [references/writing-tests.md](references/writing-tests.md) | +| Adding `agent.*` steps, picking a model, cutting cost, budgets, the replay cache | `agent` | [references/agent.md](references/agent.md) | +| Running tests: CLI flags, CI, reporters, trace pages, `report.json` for scripts, exit codes | `running` | [references/running.md](references/running.md) | +| Handling a failed run: error codes and their fixes, `--headed`, `--debug`, `--ai-trace` | `debugging` | [references/debugging.md](references/debugging.md) | +| Exploring an app toward a goal without a test file: `e2e explore`, its budgets and verdict | `explore` | [references/explore.md](references/explore.md) | +| Asked to bug bash, QA, or hunt for bugs across an app or a branch | `bug-bash` | [references/bug-bash.md](references/bug-bash.md) | + +For anything the references do not cover, the full documentation ships in +the `docs/` directory of the installed `e2e` package +(`node_modules/e2e/docs` in a single-package project); a link such as +`/reference/cli` is `docs/reference/cli.mdx`. Complete projects for Vite, +Next.js, Astro, Expo, SwiftUI, Jetpack Compose, Kotlin Multiplatform, and +Flutter are in https://github.com/tester-army/e2e/tree/main/examples. + +## Quick start + +A config with one web target and a model, and a test that drives one goal +and pins the outcome: + +```ts +// e2e.config.ts +import type { E2EConfig } from 'e2e'; +import { web } from '@e2e-dev/web'; +import { gateway } from 'ai'; + +export default { + targets: [ + { + engine: web(), + app: { + url: 'http://127.0.0.1:3000', + command: { executable: 'pnpm', args: ['dev'], log: '.e2e/logs/app.log' }, + }, + }, + ], + // The model behind every agent.* step: an AI SDK instance; gateway() from 'ai' reads AI_GATEWAY_API_KEY or a Vercel OIDC token. + agents: { + default: { + model: gateway('openai/gpt-6-luna-fast'), + system: 'You are a thorough QA agent. Verify every outcome on screen.', + }, + }, +} satisfies E2EConfig; +``` + +```ts +// tests/billing.e2e.ts +import { test } from '@e2e-dev/web'; +import { expect } from 'e2e'; + +test('a member upgrades to Pro', async ({ app, agent, screen, browser }) => { + await app.open('/settings/billing'); + await agent.act('upgrade the workspace to the Pro plan'); + await expect(screen.getByRole('status')).toContainText('Pro'); + await expect(browser).toHaveURL('/settings/billing'); +}); +``` + +## Workflow: writing a test + +1. Look at what exists: `e2e.config.ts` or `e2e.config.mts`, the `tests` glob + (default `tests/**/*.e2e.ts`), `e2e` in `package.json`. Nothing there: + follow `setup`. +2. Learn the screens before writing a test: routes, labels, roles, button + text. Semantic locators need the accessible names the app renders, so read + the components, open the page with `--headed`, or drive the live app over + the registered `e2e mcp` server (topic `mcp`): `open_session`, `observe`, + and `locate` show exact names and check a locator before you write it. +3. Write `tests/.e2e.ts`. Drive the flow with `agent.act`, one goal + per call, and pin each outcome right after with `expect` or `agent.assert`. + Exact values go through `screen`: a sign-in form in a setup test, a field + that must receive one specific string, a count that must be one number. +4. Run one file: `npx e2e run tests/.e2e.ts`. Agent steps need a + model in the config and that provider's authentication (a saved + subscription login, an API key); a local endpoint may need none. Tests + without agent steps need no model. +5. Read the trace page the terminal names under each failed test + (`.e2e/results//trace.md`): every step with what it did, the cache's + decision, what the app logged, the agent's turns, and the screen at + failure. Fix the locator, the expectation, or the app. Never add a sleep. + +## Rules + +- Run the CLI as `npx e2e ...` (or `pnpm exec e2e ...`). +- The config is `export default { ... } satisfies E2EConfig` with + `import type { E2EConfig } from 'e2e'`. `targets` is required; a UI target + names an engine and declares the app beside it: `{ engine: web(), app: { url, command } }`. + A tools-only target can omit the engine and set `platform`. +- Import `test`, `describe`, the hooks, `expect`, `credentials`, and + `secrets` from `e2e`. A test that uses the `browser` fixture imports `test`, + `describe`, and the hooks from `@e2e-dev/web`: the same runtime functions, + typed with `browser`. +- Config and tests are ES modules whatever `package.json` sets as `type`. +- Locators resolve when used. Actions wait for readiness and `expect` retries + assertions. Reads such as `textContent()` fail at once on zero matches and + `count()` answers from the current screen; nothing waits for a value to + change, so use a matcher when a value has to settle. +- A locator that matches two nodes fails with `LOCATOR_AMBIGUOUS`; narrow it + (topic `writing-tests`). +- Secrets never appear in test code. Declare accounts under `credentials` and + every other sensitive value under `secrets` in the config; resolve with + `credentials.user(name).password` or `secrets.get(name)` (separate + namespaces: `secrets.get` never returns a password), and hand the opaque + `Secret` only to `fill()` or `agent.act` params. +- Agent instructions: one goal per `act`, the wording on screen, real values + in params. Judge meaning, not phrasing: `toContain('Pro')`, not an exact + sentence a model produced. +- Check each agent goal's outcome. A passing `act` with a recorded check can + be cached and replayed without model calls (topic `agent`). +- Shape the agent for this app: `context` for vocabulary the screens use, + `system` for how it works, tools for a test API, named personas under + `agents`. When a step fails, tighten the goal first, then the context, then + the agent. +- `.e2e/` is output (`report.json`, `results/`, `cache/`, `logs/`; the + config's `output` moves the report and artifacts, never `cache/` or the + app's log). Read it, never edit it. + +## Feedback + +When e2e itself gets in your way, tell the e2e team: a command or API that +broke (`bug`), docs or this skill that misled you (`docs`), or a capability +you needed and did not find (`feature`). Send it once per problem, after you +worked around it or gave up, never for failures of the app under test. + +```bash +npx e2e feedback --type bug -m "" \ + --task "" --expected "<...>" --actual "" \ + --approach "" --command "" --agent "" +``` + +Describe e2e's behavior only: never paste app content, page text, test files, +URLs of private apps, or credentials. Secret-named environment variables and +common token shapes are redacted, but do not rely on it. `--dry-run` prints +what would be sent. Tell the user you sent it and give them the reference id +it prints. diff --git a/adapters/e2e/vendor/e2e-skill/references/agent.md b/adapters/e2e/vendor/e2e-skill/references/agent.md new file mode 100644 index 0000000..d5f1e85 --- /dev/null +++ b/adapters/e2e/vendor/e2e-skill/references/agent.md @@ -0,0 +1,348 @@ +# Agent steps + +`agent` is a fixture like `screen` and the main way a test drives the app. +Each call is one bounded invocation: fresh redacted observation, deadline, +model-call budget, no shared transcript. No agent step, no model calls. + +## Contents + +- Configure a model +- act: one goal +- assert, waitFor, extract: one question +- Write instructions the model can execute +- What the model sees +- Budgets and cost +- The replay cache +- Inspect what the model did +- Make the agent yours +- Beyond the built-in agent +- In CI + +## Configure a model + +Put an AI SDK model under `agents.default`. You can use all AI SDK providers +that support tool calls and the language model specification v2 or later +(https://ai-sdk.dev/providers). Examples: `@ai-sdk/openai`, +`@ai-sdk/anthropic`, `@ai-sdk/google`, `@ai-sdk/amazon-bedrock`, +`@openrouter/ai-sdk-provider`, `ollama-ai-provider-v2`. For a server with +`/v1/chat/completions`, use `@ai-sdk/openai-compatible`. Vercel AI Gateway +reads `AI_GATEWAY_API_KEY` or, without it, a Vercel OIDC token: + +```ts +import type { E2EConfig } from 'e2e'; +import { web } from '@e2e-dev/web'; +import { gateway } from 'ai'; + +export default { + targets: [{ engine: web(), app: { url: 'http://127.0.0.1:3000' } }], + agents: { default: { model: gateway('openai/gpt-6-luna-fast') } }, +} satisfies E2EConfig; +``` + +Subscription logins and local models: topic `setup`, section "Subscriptions +and API keys". Keep `ai@^7` installed with any provider. + +Decision models pick actions from choices the executor builds: install +`@e2e-dev/decision` and put `decisionExecutor({ model: +typeSafeAi.decisionModel('jev-latest'), textModel: openrouter('inception/mercury-2.5') })` +under `executor`. `openai.decisionModel('gpt-6-luna')` from `@ai-sdk/openai` +4.0.90 or later works too and takes the `vision: true` screenshot. +Tests stay plain language with no params. The text model writes field values +and upload paths. See the shipped `docs/decision-models.mdx` or +[the online guide](https://e2e.tester.army/docs/decision-models) for setup, +gates, and limits. + +- Pass a model instance, not a string (`INVALID_CONFIG`). +- An agents entry is one plain object of `model`, `judge`, `system`, + `context`, `tools`, `maxSteps`, `maxModelCalls`, `judgmentTimeout`, + `maxObservationBytes`, `maxInputTokens`, `providerOptions`, or `executor` + for a custom brain; none inherits `model` or `context` from `default`. +- `model` drives `agent.act`. Judgments use `judge` when set, else `model`. +- A missing model for the built-in agent raises one run-level + `MODEL_UNAVAILABLE` when the first test acquires `agent`, exit 2; auth + failures surface on the first model call as `MODEL_PROVIDER_FAILED`, or as + `CONFIG_LOAD_FAILED` for a provider that reads its key when the config loads. +- `context` is what the app calls things, sent to every model call, judges + included; `system` is how the acting agent works, read only by the act + loop. Both live on the agents entry; `agentContext` on a test or group + adds more. + +### Choose an agent + +Tests use `agents.default` unless selected otherwise: + +- `e2e run --agent buyer` picks another configured agent. +- `{ agent: 'buyer' }` on a test or group pins that agent; a list such as + `{ agent: ['buyer', 'admin'] }` runs each test once per agent. +- `--agent buyer,admin` runs unpinned tests for both, narrows a pinned list + to matching names, never replacing a pin it does not name. +- `{ agent: 'name' }` on an `agent.*` call overrides the test's choice. + +For signed-in personas, pair an agent with a `session` in a describe block +repeated per persona; the cache keys every entry by the agent and its +context, so each persona records and replays its own steps. + +## act: one goal + +```ts +import { credentials } from 'e2e'; + +await agent.act('add a todo named "Buy milk" and mark it done'); +await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } }); + +const member = credentials.user('member'); +await agent.act('sign in with the given credentials', { + params: { + username: member.username, + password: member.password, // a Secret: the model sees its name, the runner fills the field + }, +}); +``` + +`act(instruction, options?)` runs a multi-action flow to a verdict. Passed +resolves with `summary`, `modelCalls`, `actions`, and `cache` (the replay +cache's part). Failed or blocked throws an `AgentError` whose `code` says +why: `ACTION_FAILED` (product failure); `STEP_BUDGET_EXHAUSTED`, +`STEP_TIMEOUT`, `CONTEXT_OVERFLOW` (out of room); +`AUTH_CREDENTIAL_UNAVAILABLE`, `AUTH_CREDENTIAL_INVALID`, +`SECRET_UNAVAILABLE`, `ENVIRONMENT_UNAVAILABLE`, `SEED_DATA_MISSING`, +`TEST_SETUP_FAILED`, `AUTOMATION_UNSUPPORTED`, `POLICY_DENIED` (blocked +from outside); +`MODEL_OUTPUT_INVALID` (unusable answer). Full list: topic debugging. + +Options: + +- `params`: values the instruction names; the runner fills a `Secret`, + and a run-unique `unique(\`E2E ${Date.now()}\`)` keeps the cache working + across runs. +- `timeout`: the config `timeout`. +- `maxSteps` (default 25 actions), `maxModelCalls` (default 25): may only + lower the agent's limits, else `INVALID_ARGUMENT`. +- No `schema` (use `extract({ schema })`) and no `vision`: the model decides + when it needs pixels. + +The tools, one per engine action. Name the target as the screen names it +(files in `params`); the agent picks the verb: + +- `observe`: re-read the screen after waiting on work in progress. +- `tap`: button, link, menu item, tab, checkbox, row, field. +- `double_tap`/`long_press`/`right_click`: second-click item, long-press + menu, context menu only. +- `hover`: menus, flyouts, tooltips that open on the pointer. +- `type`: one input; on browser/device no target means focus; `replace` clears. +- `press`: one key to a node or, on browser/device, to focus; `times` up to 20. +- `select`: one option by visible label. +- `check`: set a checkbox, switch, or radio to a state, not flip it. +- `scroll`: viewport or one scrollable node, a screen or a few. +- `scroll_to`: a listed node into view, or by `text` page a list to a row. +- `drag`: one node onto another. +- `upload`: project-root files to a file input; outside it or hidden (`.env`) + is `POLICY_DENIED`. +- `navigate`: a URL or app-relative path. +- `back`: browser history or in-app back. +- `type_secret`: a declared secret by name; plaintext never reaches the model. + Offered only when the step declares secrets and the engine can fill them. +- `screenshot`: attach viewport pixels; every later result then carries one. +- `tap_at`/`hover_at`/`press_at`/`select_at`/`type_at`: a screenshot point. +- `dismiss_keyboard` (device): hide the on-screen keyboard. + +The focused field's selected text is listed as `selection="..."` (never for +a secret), so a repeated `Shift+ArrowLeft` selects one word with visible +feedback. Point tools serve a canvas shape, map pin, image region, or system +sheet control, hit-testing the tree first so a listed control underneath is +acted on by id; `type_at` on nothing listed taps, then types. A screen with +nothing to tap by id opens with a screenshot attached. A device has no +`right_click`, `select`, `upload`, or `scroll_to` by node id. Pixels are +masked and withheld after a secret fill (topic writing-tests): act on pixels +before signing in, or in a test of its own. A result says when an action +closed an on-screen keyboard; on a touch screen that tap was often spent +closing it, so act again. + +## assert, waitFor, extract: one question + +```ts +import { z } from 'zod'; + +await agent.assert('the dashboard shows a trial badge'); // one look, one judgment + +await agent.waitFor('the export finished and a download link appeared', { // polls + interval: 500, + timeout: 120_000, +}); + +const data = await agent.extract('every todo title and how many remain', { // structured output + schema: z.object({ titles: z.array(z.string()), remaining: z.number().int() }), +}); +expect(data.titles).toContain('Buy milk'); +``` + +- `assert` does not poll. False is `ASSERTION_FAILED` with the model's + explanation and a screenshot in the report; too little on screen to + decide is `ASSERTION_INCONCLUSIVE`, also a failure, so reach the right + screen first and ask about what is visible. Judged from the tree alone, + it adds `pass vision: true when the answer is in pixels`, as does + `waitFor`'s timeout after an inconclusive round. +- Judgments see the assertion and the current screen only, never prior + steps or the act loop's summaries; malformed output gets one repair + round, then `MODEL_OUTPUT_INVALID`. +- A value shown in more than one place (a total in the summary and on the + pay button) must agree everywhere, or the judgment fails. Name the one + you mean (`'the order summary total is $42.00'`) when only it matters. +- `waitFor` observes every `interval` (default 3 s), judges only when the + screen changed, and is `STEP_TIMEOUT` after `timeout` (default 30 s). +- `extract` takes any Standard Schema validator (zod works); the model sees + the schema's shape, never its value rules (`min`, `max`, lengths, + patterns), which check what it read. Data the screen does not show is + `ASSERTION_INCONCLUSIVE` naming what was missing, never `""` or `0`; to + accept absence, ask for it (`'the phone, or null when none is shown'` + with `.nullable()`). + +`vision` on a judgment picks the evidence: `false` (default) the tree; +`true` the tree plus a masked screenshot; `'only'` the screenshot alone, for +what the screen presents (an overlay, a broken layout, a chart) where the +tree would answer first. `'only'` never falls back to the tree: unprovably +masked pixels fail with `POLICY_DENIED`. + +## Write instructions the model can execute + +- One goal per `act`; goal order is the test's, the path the model's. +- Use the words on screen: `'open the Billing tab'`, not `'upgrade'`. +- Values go in `params`, never expanded: `act('rename to {name}', { params })`. +- Do not describe mechanics the runner handles: waiting, scrolling, retries. +- Pin every `act` outcome right after it; that check lets the cache record: + +```ts +await agent.act('create a workspace named "Atlas" on the Pro plan'); +await expect(screen.getByRole('status')).toHaveText('Created "Atlas" on the Pro plan'); +``` + +The model gets the instruction verbatim plus the params as a separate +block; the check also makes the test model-portable. Off-screen state (a +database row) can land after `act` returns; `expect.poll` the read instead +of sleeping. + +## What the model sees + +A redacted snapshot of the screen (roles, names, text, states), prior-step +summaries, and your context; never raw HTML, cookies, headers, environment +values, or a `Secret`'s value; password fields masked. The first +screen of a step arrives whole; later action results report what changed, +keyed by node ids stable while an element exists, or the whole screen when +most changed. Pixels arrive through `vision` on a judgment or the act loop's +`screenshot` and point tools, masked and withheld after a secret fill. When +the browser engine's tree capture times out, the model gets a screenshot and +a warning, a judgment needs `vision: true`, and no control may be inferred +absent nor old node ids reused. Nothing the model returns runs as code or +selectors: the runner validates and authorizes every tool call first. + +## Budgets and cost + +| Call | Model calls | Default timeout | +| --- | ---: | --- | +| `act` | up to `agents..maxModelCalls` (25) | the config `timeout`, 120 s | +| `assert` | 2 | 30 s | +| `extract` | 2 | 30 s | +| `waitFor` | up to `agents..maxModelCalls` (25) | 30 s | + +- Slow model calls: raise the step or test `timeout` for `act`, the agent's + `judgmentTimeout` for judgments, `actionTimeout` for slow UI. + `STEP_TIMEOUT` and `STEP_BUDGET_EXHAUSTED` fail the test; smaller goals + help. +- `--debug` prints phase timings and a per-step table (duration, model + calls, tokens, cache share, cost) to stderr and saves step transcripts as + artifacts. + +## The replay cache + +A passing `agent.act` saves its actions once a later check verifies the +outcome; the next run replays them without model calls, and the live agent +continues from the current screen when the app or final state no longer +matches. Misses and hand-offs use the model; `agent.assert`, +`agent.waitFor`, and `agent.extract` are never cached. + +- On by default (`read-write`), `read-only` in CI, off via `cache: 'off'` + or `--no-cache`. Entries live in `.e2e/cache/`; deleting the directory + only slows the next run. +- An entry is written only after a later verification passes (a locator or + engine `expect` matcher, `locator.waitFor`, `browser.waitForURL`, + `agent.assert`, `agent.waitFor`), so an unchecked `act` never replays; a + plain-value `expect`, `expect.poll`, `agent.extract`, another `act`, or + the attempt passing confirms nothing. +- A replay needs the app on the recorded route (origin, path, and query, + ids and tokens aside; unless the recording opens with a navigation), + re-finds each control by role, name, test id, placeholder, input purpose, + and its named row (an unnamed control with twins and no named container + hands off), and passes alone only when the recorded end route is back, + every control that appeared (with its checked or selected state) is + there, every one that went away is gone, at least one of those changed + during the replay, and no new alert showed; otherwise the agent takes + over mid-step. Entries are keyed per agent and per `agentContext`. `step.cache.reason` says why: `no-entry`, + `wrong-context`, `target-not-found`, `target-ambiguous`, `end-mismatch`, + and so on. +- A step recording no actions, or changing nothing on screen or in the + route, creates no entry; one whose `unique()` value + equals, is spelled inside, or is the encoded form of another param's value + is not recorded either (`step.cache.notRecorded`: `param-collision`). +- `e2e init` gitignores `.e2e/cache/`; remove that line to commit entries + and share replays with CI and teammates (CI stays `read-only` unless + `cache: 'read-write'` is set). +- A failing run evicts the entries it implicates; `--no-cache` rules the + cache out of a failure. A cancelled run and a failure where no model + answered (`MODEL_PROVIDER_FAILED`, `MODEL_UNAVAILABLE`) implicate nothing + and evict nothing. +- With committed recordings, `--strict-cache` in CI fails a recording that + no longer replays with `REPLAY_STALE` instead of quietly spending model + calls every run; re-record with a `read-write` run without the flag and + with `cache.strict` off, then commit. Unrecorded steps still run live; + retries replay too, and a strict run never writes the cache. + +## Inspect what the model did + +```bash +npx e2e run tests/checkout.e2e.ts --debug # step table, transcripts as artifacts +npx e2e run tests/checkout.e2e.ts --ai-trace # writes /ai-trace.json (.e2e/ai-trace.json by default) +npx unbox-ai runs .e2e/ai-trace.json # one line per agent step +npx unbox-ai summary .e2e/ai-trace.json --run 0 # turns, tokens, tool calls of one step +``` + +Never read the trace directly: megabytes of resent context, images replaced +by byte counts. `--no-cache` traces the whole flow, since replayed steps +make no model calls. + +## Make the agent yours + +1. **The goal.** A failed step usually named something the screen does not; + reword it with on-screen labels, and `--debug` shows what the model saw + and tried. +2. **`context`.** Vocabulary every step needs (plan names, what a + "workspace" is, which tab holds billing), once on `agents..context` + or per test via `agentContext`. +3. **`system` on the agent.** How carefully it verifies, what it never + does, how it treats a modal; a UX reviewer, a cautious QA persona, and a + fast smoke agent are three `system` prompts on one model. +4. **Tools.** A test API the agent may call mid-flow (seed a cart, mint a + coupon) via `tools`; see below. +5. **The model and its options.** `providerOptions` for reasoning effort, + or another model for one persona. `--agent ` runs the suite as any + configured agent, comparing candidates on the same tests; every result + records which agent ran it. + +## Beyond the built-in agent + +- `tools: { seedCart }` on an agents entry adds AI SDK tools wrapped with + `defineTool(tool({ ... }), { mutates: true })` from `e2e/agent` for a + test API a flow calls mid-step. +- `createToolLoopExecutor` keeps the loop and replaces prompt and tool + vocabulary. +- Any `StepExecutor` (`{ name, version?, cache?, runStep(ctx) }`) goes under + `executor`; the runner still owns observations, actions, budgets, and the + report, and `system` or `tools` beside `executor` is `INVALID_CONFIG`. + +Reference: https://e2e.tester.army/docs/agents + +## In CI + +Run the suite, agent steps included, on every pull request, passing the key +the config's model reads (`env: { AI_GATEWAY_API_KEY }` for `gateway()`) +from secrets. Sharing `.e2e/cache/` lets CI replay verified action steps +without model calls; CI defaults (`read-only` cache, retries): topic running. diff --git a/adapters/e2e/vendor/e2e-skill/references/bug-bash.md b/adapters/e2e/vendor/e2e-skill/references/bug-bash.md new file mode 100644 index 0000000..62b491e --- /dev/null +++ b/adapters/e2e/vendor/e2e-skill/references/bug-bash.md @@ -0,0 +1,247 @@ +# Running a bug bash + +A bug bash is many `e2e explore` runs at once, one charter each, then a +verification pass that turns every claimed bug into a repro test that fails +for the reason reported. Plan the charters, fan them out, merge the findings, +prove each bug. Report confirmed bugs only, each with its failing test. + +## Contents + +- 1. Prepare +- 2. Plan charters +- 3. Fan out +- 4. Merge +- 5. Triage +- 6. Verify +- 7. Report +- Rules + +## 1. Prepare + +- A config with a target and an agent that holds a model (topic `setup`), + and authentication for its provider. +- The app must serve several explorers at once: start it once and set + `reuseExisting: true` on the target's `app.command`, or declare the URL with + port `0` so each run starts its own app on a free port. `reuseExisting` is + ignored when `CI` is set, as in many agent sandboxes: use port `0` or + unset `CI`. A fixed port that is not reused fails every run after the first. +- When the target's `app.command` brings up a stack of its own (a database + it starts and removes on exit), the first explorer to finish removes the + database under the rest. Start the stack once yourself with the config's + command and environment, and explore with a bug-bash config that declares + no command (below). +- Explore a production build when the project has one. A dev server that + compiles a route on first visit reads to the explorer as a dead link. +- Seed one disposable workspace or user per charter with the project's own + fixtures or seed scripts and declare each as a credential (`bb-`). + Explorers that share an account report each other's edits as bugs. +- Start signed-in charters from a saved session: `e2e explore --session + ` runs the setup test that saves it and explores signed in (topic + `explore`). A setup that types a password withholds screenshots from then + on; sign in by cookie or API call to keep them (topic `writing-tests`). + Each charter with its own account needs its own setup and session. +- Put what the local app cannot do in the agent's `context`: integrations + without keys, what is seed data, what must never be clicked (paid runs, + real accounts). Add the explorer's own blind spots from step 5's artifact + bucket (the config below carries the sentence); without them those + families dominate the findings. +- A bash against a deployed site others use is read-only: no signups, + sign-ins, or submissions, nothing injection-shaped in URLs, no request + loops. A WAF block is the firewall working, not a finding, and it can + follow the runner's IP into every later charter. +- Each exploration step needs 40 or more `maxSteps` and `maxModelCalls` on + the agent; a config tuned for test steps (`maxSteps: 15`) starves it. That + is per step, separate from `--max-steps`, which counts a charter's steps. +- On a mobile target, declare one target per device and give each explorer + and verifier its own (topic `setup`). +- Skip `--headed`. On an engine that records, `--video` gives each run a + replay for any confirmed bug. + +A bug-bash config, left untracked, spreads the project's config and +overrides what a bug bash needs. Typing the import as `E2EConfig` keeps +`shared.tests` and `shared.credentials` compiling when the base declares +neither: + +```ts +// e2e.bugbash.config.ts +import type { E2EConfig } from 'e2e'; +import { web } from '@e2e-dev/web'; +import { gateway } from 'ai'; +import base from './e2e.config.ts'; + +const shared: E2EConfig = base; + +// What the local app cannot do, plus the explorer's blind spots (step 5's artifact bucket). +const context = + 'Sign in with the credential the goal names. The local app sends no email and has no AI key. Never start a paid run or connect an integration. ' + + 'Not bugs: a link that opens a new tab leaves this one unchanged; accessible text splits around inline links, so judge copy by the rendered screen when a screenshot is available and never report split text alone as broken copy; an infinite-scroll "Loading more" sentinel loads when scrolled into view; images lazy-load, so scroll and wait before calling one blank.'; +const persona = { model: gateway('openai/gpt-6-luna-fast'), maxSteps: 40, maxModelCalls: 40, context }; + +export default { + ...shared, + // The project's tests, so a repro can use its setup tests' sessions, plus the repro tests from step 6. + tests: [shared.tests ?? 'tests/**/*.e2e.ts', 'tests/bugbash/**/*.e2e.ts'].flat(), + // The app already runs: no command. + targets: [{ name: 'web', engine: web(), app: { url: 'http://127.0.0.1:3000' } }], + retries: 0, + reporters: ['list'], + credentials: { + ...shared.credentials, + // The seed script's password, from the environment rather than the file. + 'bb-cart': { username: 'bb-cart@example.test', password: process.env.BUGBASH_PASSWORD ?? '' }, + 'bb-account': { username: 'bb-account@example.test', password: process.env.BUGBASH_PASSWORD ?? '' }, + }, + // The postures from step 2 as personas: same model and budgets, a different stance; each charter picks one with --agent (step 3). + agents: { + default: persona, + skeptic: { ...persona, system: 'Distrust every number, date, count, and claim on screen; cross-check each against every other place it appears.' }, + fuzzer: { ...persona, system: "At every input, run the goal's input matrix before anything else, judging each entry before the next. Never take the happy path." }, + }, +} satisfies E2EConfig; +``` + +Set `BUGBASH_PASSWORD` to 6 or more characters first, or the config fails to +load. Pass `--config e2e.bugbash.config.ts` to every command below, and +`open_session {config: "e2e.bugbash.config.ts"}` over MCP. + +## 2. Plan charters + +A charter is one `e2e explore` goal: one area, one posture, one sentence, +naming the start route and, when needed, the credential (`Sign in as +credential bb-cart. Starting at /cart, ...`). Read the routes, navigation, +and forms first; for a branch, `git diff --stat` against the base. + +| Posture | Charter shape | +| --- | --- | +| First-time user | `Starting at /signup, sign up and complete onboarding like a first-time user; report anything confusing, broken, or inconsistent` | +| Numbers and copy | `Starting at /cart, change quantities and apply a coupon; check every price, total, and label against the rest of the page` | +| Edge input | `Starting at /settings/profile, submit each field empty, too long, with unicode and with leading spaces; report validation that is missing or wrong` | +| State | `Starting at /projects, create, rename, and delete a project, reloading and going back after each; report state that is lost or stale` | +| Error paths | `Starting at /login, try a wrong password, an unknown account, and a locked account; report errors that are missing, misleading, or leak detail` | + +Aim for five to ten charters, each with its own slug. Overlap is fine; +duplicates merge in step 4. Give each posture its persona (config above), +picked per charter in step 3: a generic agent walks past a stat that +contradicts the same stat on another page, the skeptic catches it. An +edge-input charter names its exact matrix (empty, a 300-character string, +unicode, leading spaces, literal special characters) or it spends the whole +time budget before judging a single result. + +## 3. Fan out + +One `e2e explore` per charter with its own output directory: +`--output .e2e/bugbash/` writes `report.json` and `results/` there, +and `--reporter list,markdown` adds `summary.md`. `e2e init` gitignores +specific `.e2e/` paths, not `.e2e/bugbash/`: add it to `.gitignore` or +delete it when done, and delete it before a new bug bash. `charters.txt` and +the logs under it are yours; the rest is the runner's output, read only. + +Run them as background shell jobs, four at a time, not as subagents. One +line per charter, `slug|target|agent|charter`, the agent naming its persona, +then (`xargs -0 -P` is a GNU and BSD extension, present on macOS and Linux): + +```bash +mkdir -p .e2e/bugbash +cat > .e2e/bugbash/charters.txt <<'CHARTERS' +cart|web|skeptic|Starting at /cart, change quantities and apply a coupon; check every price, total, and label against the rest of the page +account|web|fuzzer|Starting at /settings/profile, submit each field empty, a 300-character value, unicode, and leading spaces; report validation that is missing or wrong +CHARTERS +while IFS='|' read -r slug target agent charter; do + [ -n "$slug" ] && printf '%s\0%s\0%s\0%s\0' "$slug" "$target" "$agent" "$charter" +done < .e2e/bugbash/charters.txt | xargs -0 -n 4 -P 4 sh -c \ + 'npx e2e explore "$4" --config e2e.bugbash.config.ts --target "$2" --agent "$3" --output ".e2e/bugbash/$1" --max-steps 6 --video --reporter list,markdown < /dev/null > ".e2e/bugbash/$1.log" 2>&1' _ +``` + +The log's summary prints `AI` (cost) and `Duration`. Exit code `1` means +issues were reported: read the log either way. Exit codes `2` and `3` are +setup and environment problems; fix them and rerun that charter alone. + +## 4. Merge + +Read each log's `Findings` section (topic `explore`); `summary.md` holds the +same, and `report.json` has the record under `run.explore`. The video is in +the attempt's `video/` directory under `.e2e/bugbash//results/`. + +Merge findings that describe one defect: same path, same broken behavior. +Keep the clearest reproduction and every charter that hit it. Keep warnings +in a separate list unless the user asked for polish. + +A charter that filled a password, or whose session's setup did, prints no +`evidence` line after that point. Its video is not masked: check it for +secrets before sharing (topic `writing-tests`). + +## 5. Triage + +Sort every finding before writing any test. Read the source to sort them. + +| Bucket | Sign | Outcome | +| --- | --- | --- | +| Explorer artifact | A "dead" `target="_blank"` link whose destination opens when clicked with popup capture or navigated to directly, a broken sentence the screenshot renders whole, a "Loading more" sentinel nothing scrolled to, a lazy-loading image or embed | Rejected with the check that settled it; settle this bucket first. A new-tab link whose destination never opens stays a candidate | +| Environment | Fails on a key, a service, or a limit only the local stack lacks (an email provider, an AI key, a billing plan) | Rejected, naming the variable or service; note separately when the app handles the failure badly in a way production users would see, such as showing the raw error | +| Design | The code, its tests, or its copy say the behavior is intended | Rejected, citing where | +| Fixture | The seed data lacks a field real records always have | Rejected, naming the field | +| Candidate | None of the above | Verify it (step 6) | + +## 6. Verify + +A finding is a model's claim; prove each candidate before reporting it. When +your client can start subagents (Claude Code's Agent or Task tool, for one), +start one per area with its three to five candidates as the log printed +them and these steps, up to four at a time, the `e2e mcp` server's default +session limit. Each reports back, per finding: confirmed or rejected, the +root cause as `file:line` when it may read the source, the repro test path, +and the failure it saw. Without subagents, verify one area after another. + +1. Read `actual` against the screenshot, or the video when there is none. + A finding the evidence contradicts is rejected here. Settle the artifact + bucket first: a "dead" link's destination must actually open (a valid + href with a prevented default is still dead), a copy claim must show in + the rendered screenshot. +2. Write a repro test that follows the reproduction and asserts the + expected behavior, so it fails today and passes once the bug is fixed. + Put it under `bugbash/` inside the directory the config's `tests` glob + covers (`tests/bugbash/.e2e.ts` for the default): a file the glob + does not match is never selected, whatever path you pass to `e2e run`. + Build on the project's fixtures (a `test.extend` fixture, a setup test's + session). Prefer `screen` actions and `expect` with exact values; + `agent.act` for a step that varies, `agent.assert` for an outcome only + judgment can check (topic `writing-tests`). Tag it `{ tags: ['bugbash'] }`. +3. Get exact locators from the live app: with the `e2e mcp` server + registered, `open_session` with the bug-bash config, pass its session id + to every call, walk the reproduction, and `locate` each locator before + writing it (topic `mcp`). Close your session when done. +4. Run the file alone: `npx e2e run tests/bugbash/.e2e.ts`. Confirmed + only when it fails with `ASSERTION_FAILED` on the assertion that encodes + the bug. Any other failure (`LOCATOR_NOT_FOUND`, a timeout, a setup + error) means the test is wrong: fix it and rerun. A passing test means + the bug did not reproduce: reject the finding, say so, and move the test + out of `tests/bugbash/` (to `.e2e/bugbash/extra-tests/`, or offer it as + a regression test). Only failing repro tests stay there. + +## 7. Report + +Lead with the confirmed bugs, most severe first. For each: title, path, one +line of expected against actual, root cause when known, steps, screenshot +and video paths, repro test, and the charters that found it. Then the +environment candidates that are production risks, marked unverified. Then +the rejected findings grouped by reason (environment, design, fixture, did +not reproduce), and the warnings. End with the charters run, their cost, and +the areas no charter reached. + +The repro tests fail until the bugs are fixed: where the project's `tests` +glob covers `tests/bugbash/`, its gating run leaves them out +(`npx e2e run --exclude-tag bugbash`) or they stay uncommitted. Offer to fix +each bug: the repro test turning green is the proof, and it stays as the +regression test with its `bugbash` tag removed. + +## Rules + +- Never report an unverified finding as a bug. "The explorer reported" is + not "confirmed". +- One charter, one area. A charter that spans the whole app ends at its + step budget having skimmed everything. +- Seeded data and test accounts the app ships for development are not + bugs; say so in the agent's `context`. +- Leave the project's files as you found them: the bug-bash config, seed + script, and repro tests stay untracked until the user asks otherwise, and + stop the stack you started when the user is done. diff --git a/adapters/e2e/vendor/e2e-skill/references/debugging.md b/adapters/e2e/vendor/e2e-skill/references/debugging.md new file mode 100644 index 0000000..fcbeb7a --- /dev/null +++ b/adapters/e2e/vendor/e2e-skill/references/debugging.md @@ -0,0 +1,144 @@ +# Debugging a failing run + +Paths below sit under the configured `output` directory; `.e2e` is the +default. + +## Contents + +- Read the failure +- Error codes +- Tools +- Flaky tests +- Is it the app? + +## Read the failure + +1. Every failure in the `list` output ends with `❯ trace + .e2e/results//trace.md`. Open that trace page: `Look at:` (the line it unwound + through), `Expected:` / `Observed:` (an `expect`), `Asked for:` and + `Waited:` (a locator), whether every attempt failed alike (a bug, not a + flake), then every step with what it did under it: + - the node an action landed on (`tap button "Save" (12ms)`), or `✗` and + its code; + - what an assertion read while it waited (`expect gave up after 16 reads + in 1.6s: text "0 remaining" (1 match) x16`); one value throughout + points at a wrong expectation, a broken app, or one slower than the + timeout, changing values at a race; + - what the app logged meanwhile (`✗ network error: POST /api/save 500`, + `✗ uncaught error: ...`, `⚠ console warning: ...`, `ℹ console: ...`); + `❯ app 2 errors` in the terminal says there are some, and the page's + `## App log` lists up to 50 with their step, errors and warnings + first, counting the rest; + - where the page went (`↪ navigated to /login`, a new tab the app + opened, a frame that loaded); + - how the screen changed since the step before (`screen: 2 changes since + step 3` with the added, removed, and changed nodes); `unchanged` after + an action means it had no visible effect; + - the cache's decision for an agent step: replayed, handed off and where + (`at action 2 of 3, tap button "Apply": ...`), what became of the + recording, and the entry file; + - the agent's last turns, loop-guard notes included, and the + accessibility tree at failure, one node per line. + Fix from what was there. `trace` (default `retain-on-failure`) chooses + which tests keep a page; `--trace` keeps one for every test of a run. +2. `.e2e/report.json` holds the same facts for scripts, not for reading: + each step's `events` (`engine`, `poll`, `navigation`) and the attempt's + `appLog` (each line's `step` is a step `index`) are what the page's step + lines are rendered from, before it filters and caps them. +3. When a script needs them: + +```bash +jq '.run | {status, exitCode, errors}' .e2e/report.json +jq '.run.results[] | select(.selected and .status != "passed") | {titlePath, file, status}' .e2e/report.json +jq '.run.results[] | select(.selected and .status != "passed") | .attempts[-1] + | {status, error, failure, steps: [.steps[] | select(.status != "passed") | {api, label, source, status, error}], artifacts}' .e2e/report.json +``` + + `error.details` holds the facts, `error.source` the line, `failure` the + `url`, `screen` and `screenshot` artifact ids, and `candidates`; a + failed agent step has `turns`; `selected` drops filtered-out tests + (recorded as `skipped`). +4. Artifacts, under `.e2e/results//attempt-/` (`rerun-/attempt-/` + for a `--last-failed` rerun), beside the + trace page: `screen-at-failure.txt` + and the engine's screenshot per failed attempt; downloads; with + `--video` the recording (`video/video.webm` in a local browser, each + later page `video/video-part.webm` with its own `startedAt`; + `video/video.mp4` on a device; a provider's file or link); with + `--debug` every agent step's transcript. + +## Error codes + +| Code | Usual cause | Fix | +| --- | --- | --- | +| `CONFIG_LOAD_FAILED` | The config throws while loading (a refused engine option is `INVALID_CONFIG` instead) or imports a missing package, subpath, or removed export such as `defineConfig` | Install the dependency, or fix the import or line quoted | +| `INVALID_CONFIG`, `INVALID_GLOB` | Unknown or foreign key (`app`, `webServer`, `use`, `projects`, `baseURL`), `json` with `list`; a `tests` glob with braces, classes, an absolute path, or no wildcard | The message names the key or glob to write; the app is declared in the target's `app` | +| `CONFIG_NOT_FOUND`, `CONFIG_AMBIGUOUS` | Wrong `--config` path; both `.ts` and `.mts` present | Fix the path; keep one | +| `NO_TESTS` | The glob, a positional, or a filter matched nothing; the message names each empty positional and each undeclared `--tag` with the nearest declared one | Check the config `tests`, the `.e2e.ts` suffix, the tag names | +| `NO_LAST_RUN` | `--last-failed` found no `.e2e/report.json` | Run once without it | +| `COLLECTION_ERROR` | `async` describe body, `test.setup` inside `describe`, an option forbidden in a serial group, registration outside collection | Restructure per `writing-tests` | +| `HOOK_FAILED` | `beforeAll` or `afterAll` threw; its scope's tests skip | Fix the hook; the report carries its error | +| `UNSUPPORTED_ARTIFACT` | A test's or target's `video` on an engine that cannot record | Drop it there, or set it at the config root or CLI (such targets skip with a notice) | +| `BROWSER_INSTALL_FAILED`, `LAUNCH_TIMEOUT` | Browser download failed; engine init or attempt start exceeded `launchTimeout` | Run the quoted `npx @e2e-dev/web install ` or `pnpm exec e2e-web install ` (`--with-deps` on bare Linux); raise the root `launchTimeout` (60 s default) | +| `APP_UNREACHABLE` | `app.command` never answered `readyUrl` within `startupTimeout`; on a device, a message naming the iOS automation runner: the runner failed, not the app | Read the quoted log lines; check the port, `app.url`, `app.command.env`, `app.command.startupTimeout`. Runner: rerun, else `npx agent-device daemon stop` and reboot the simulator | +| `APP_ALREADY_RUNNING` | Something already serves `url` when `command` should start | Stop it, or `reuseExisting: true` locally | +| `APP_URL_REQUIRED`, `APP_NOT_OPEN` | A navigation on a target without `app.url`; a `screen` call before `app.open()` | Add `app.url` to the target; open the app first | +| `INVALID_ARGUMENT`, `INVALID_LOCATOR` | A step argument failed validation; a locator got a bad option or filter key | Fix the call the code frame names | +| `LOCATOR_NOT_FOUND` | Wrong role or name, inexact text, element off screen or in an iframe, page not open | Read the markup for the accessible name; `exact: false` or a RegExp; `browser.frameLocator` for iframes; `app.open()` first; `--headed` | +| `LOCATOR_AMBIGUOUS` | Two matches (hidden duplicate, repeated label) | `{ name }`, a container scope, `filter`, `first()`, or `{ visible: true }` | +| `ASSERTION_FAILED` | Wrong expectation, or the state settles later than 5 s; for `agent.assert`, a false judgment (explained in the report) | Compare with the report's actual text or the screenshot; `{ timeout }` on the matcher; rewrite the question | +| `ASSERTION_INCONCLUSIVE` | `agent.assert` or `agent.extract` asked about what the screen does not show (another page, still loading, only in pixels); a failure, never a pass | `app.open()` or `agent.waitFor` the right screen first; ask about what is shown; `vision: true` when the answer is in pixels (the message says so) | +| `MODEL_OUTPUT_INVALID` | An `extract` or `assert` answer failed the schema after one repair round | Simplify the schema or question; pick a stronger model | +| `ACTION_FAILED` | Element not actionable (covered, disabled, detached), or an operation timed out | `expect` the condition first; close overlays; check `actionTimeout` | +| `TEST_TIMEOUT` | The attempt exceeded `timeout` (120 s) | Split the test, or raise `timeout` | +| `STEP_NOT_AWAITED` | The body returned while a step or `expect.poll` still ran, or an `afterEach` hook or fixture teardown returned while an `expect.poll` still ran: a call without `await` | `await` the call the code frame names (every `app`, `agent`, `screen`, `expect`, `expect.poll` call) | +| `MODEL_UNAVAILABLE` | `agents..model` holds no AI SDK instance: reported once under `run.errors` at the first `agent` fixture; the run stops. A model-less custom executor fails only `waitFor` and `extract` this way | Construct one, e.g. `gateway('openai/gpt-6-luna-fast')` from `ai`, and export its key (`AI_GATEWAY_API_KEY`) | +| `MODEL_PROVIDER_FAILED` | Network, 5xx, rate limit, no credits, or no response in 120 s, after the transport retries | Check the credential (the key; for keyless `gateway()` the Vercel CLI login and `.vercel/project.json`) and quota; retry (exit 3) | +| `REPLAY_STALE` | `--strict-cache` or `cache.strict`, and a committed recording no longer replays; `step.cache.reason` says why | Rerun read-write with the knob the message names off; commit the changed entry under the directory it names (a custom `cache.store` is written by that run) | +| `AUTOMATION_UNSUPPORTED` | The step needs an interaction the engine's toolset lacks (drag on a device, say); blocked, exit 1 | Do that step with `screen` actions, or run the test on a target that supports it | +| `STEP_TIMEOUT`, `STEP_BUDGET_EXHAUSTED` | Goal too big or ambiguous, or a slow provider | Split the goal, use on-screen wording, add vocabulary via `agents..context` or `agentContext`; raise `maxSteps` or `maxModelCalls` (budget), the config `timeout` (`act`), `judgmentTimeout` (judgments); `--debug` | +| `CONTEXT_OVERFLOW` | Screen plus step history did not fit the model's context window (`act` after one shrink-and-retry; a judgment on the first overflow) | Lower `agents..maxObservationBytes`, split the step, or pick a larger-window model | +| `POLICY_DENIED` | A URL whose scheme is not `http:` or `https:` (`file:`, `view-source:`, `data:`); a password `Secret` into a non-password sink; reading a secure field, `toHaveValue`, `toHaveText`, `toContainText`, `toHaveAttribute` included; `app.screenshot()` after a secret fill, or an `act` or `vision: true` step whose tree read timed out after one | http(s) or `about:blank` only; passwords into password inputs only; assert the outcome, not the value; screenshot before filling secrets | +| `UNSUPPORTED_CAPABILITY` | A fixture the engine lacks (`browser` on a device), `schema` or `vision` on `act`, an action the surface lacks | Declare `requires: ['browser']`; drop the option or action | +| `SESSION_UNAVAILABLE`, `SESSION_CONTRACT` | `session: 'x'` with no setup saving `x`; a setup that skipped a declared name. `SESSION_MISMATCH`, `SESSION_EXPIRED`, `SESSION_INVALID`: the stored session is another run's or app's, expired, or corrupt | Add or fix the `test.setup`; rerun it | +| `ONLY_IN_CI` | `test.only` reached CI | Remove it | +| `AUTH_CREDENTIAL_UNAVAILABLE`, `AUTH_CREDENTIAL_INVALID` | `credentials.user('x')` for an undeclared name; the app rejected a configured credential | Add it to `config.credentials`; fix the stored value | +| `SECRET_UNAVAILABLE` | `secrets.get('x')` for an undeclared name | Add it to `config.secrets`; `E2E_SECRET_X` only overrides a declared one | + +## Tools + +| Do | When | +| --- | --- | +| `--headed` | Watch the failing step | +| `--workers 1 --retries 0` | Take parallelism and retries out of the picture | +| `--no-cache` | Rule out a stale `agent.act` replay | +| `--debug` | Each agent step's duration, model calls, cost, transcript | +| `--ai-trace`, then `npx unbox-ai runs .e2e/ai-trace.json` | What the model saw and called | +| `--video`, `--video=retain-on-failure` | Watch the failed attempt; `step.startedAt` minus the segment's `startedAt` is the step's offset into it | +| `command.log: '.e2e/logs/app.log'` | The app's output when it never gets ready or errors mid-test | +| `await app.screenshot('before-submit')` | Evidence before a secret is filled; later calls fail with `POLICY_DENIED` | +| `CI=1 npx e2e run` | Reproduce CI-only behaviour (defaults: topic running) | + +## Flaky tests + +- A read (`textContent()`, `count()`) caught a value mid-update: use a + matcher. +- Shared data: unique names per run, `afterEach` cleanup, or a `serial` + group. +- App not ready: assert on the element you are about to use, not the + previous page. +- An agent judgment asserts exact phrasing: judge the fact, add a + deterministic `expect` beside it. +- Load timing: `retries` masks it; `--workers 1 --headed` shows it. +- Measure a fix: `npx e2e run --repeat-each 10 --retries 0` + (`--no-cache` for agent steps). `Repeats` reads `0 of 1 test passed all + 10 runs` over `3/10 passed · repeat 0 ASSERTION_FAILED · ...` (0-based) + before, `1 of 1 test passed all 10 runs` after. + +## Is it the app? + +A deterministic step failing every run at the same place with the same code +is a product bug or a changed screen, not a flake: reproduce once with +`--headed`, then fix the app or update locator and expectation together. A +blocked agent step (`AUTH_CREDENTIAL_UNAVAILABLE`, `ENVIRONMENT_UNAVAILABLE`, +`SEED_DATA_MISSING`) exits 2 or 3 on purpose: fix the environment. diff --git a/adapters/e2e/vendor/e2e-skill/references/explore.md b/adapters/e2e/vendor/e2e-skill/references/explore.md new file mode 100644 index 0000000..94e5f07 --- /dev/null +++ b/adapters/e2e/vendor/e2e-skill/references/explore.md @@ -0,0 +1,136 @@ +# Exploring without a test + +`e2e explore` runs the agent against the app with a goal instead of a test +file: see what it can do with an app before tests exist, hunt for regressions +on a branch, or find what is worth turning into a test. It needs a config with +a target and an agent that holds a model, nothing else. + +```bash +npx e2e explore # goal: "Explore the app and find bugs" +npx e2e explore 'Explore checkout like a first-time buyer and report anything off' +npx e2e explore --target web --max-steps 4 --headed +npx e2e explore 'Hunt for broken forms' --video +npx e2e explore --session admin 'Explore the admin settings' +``` + +## Contents + +- What a run does +- Start signed in +- Flags +- Reading the result +- When it does not fit + +## What a run does + +1. Opens the app on the target's URL. +2. Plans one step: a structured model call reads the goal, the steps and + findings so far, and the current screen, and answers with a title and a + concrete charter for one flow, or decides the goal is covered. +3. Runs the charter as an `agent.act()` step with the project's tools plus + `report_finding`. The agent reports each defect the moment it has evidence: + title, `issue` or `warning`, severity 1 to 5, expected, actual, reproduction + steps. The runner adds the path and a redacted screenshot. +4. Repeats until the planner finishes, the step limit, the clock, or three + failed or blocked steps in a row that reported nothing; then asks for a + closing assessment. + +A failed step does not end the run: it is recorded, and only reported issues +fail the product verdict. A run whose charters were all blocked stays blocked. +A step that hits its action or time budget ended at its limit and counts as +neither. Configured `credentials` reach the explorer as step secrets: the +planner knows the account names and usernames, and the agent fills passwords +with `type_secret` by name. That tool is offered only when the engine can fill +secrets; otherwise the agent skips signing in and says so in the step summary. + +## Start signed in + +`--session ` starts from a session a setup test saves +(`test.setup('...', { sessions: ['admin'] }, ...)` and `session.save('admin')`, +topic `writing-tests`). The run collects the config's test files, runs exactly +the setup that declares the session, then restores it into the exploration +and, on a target with a URL, opens the app, as a test with +`{ session: 'admin' }` does. No other test runs. The setup runs as for +`e2e run`, with the configured agents, cache, and retries; only the +exploration runs as the explorer. The planner and the agent are told they +start signed in, so no charter is spent signing in again. A name no setup +declares fails before any app process starts with `COLLECTION_ERROR`, naming +the declared sessions. + +A setup that filled a secret taints the restored session, so every finding +goes without a screenshot; one that signed in without a fill (a `browser.setCookies` +session cookie, say) keeps them where the engine captures pixels, and only +configured secrets are redacted (topic `writing-tests`). + +## Flags + +| Flag | Default | Effect | +| --- | --- | --- | +| `[goal]` | `Explore the app and find bugs` | One quoted sentence: the area and the posture. | +| `--target ` | first configured target | The one target to explore. | +| `--agent ` | `default` | Build the explorer from another configured agent (`agents.`). | +| `--session ` | none | Run the setup that saves this session, then explore with it restored. | +| `--max-steps ` | 8 (1 to 12) | Exploration steps at most. | +| `--timeout ` | 600000 (180000 to 900000) | Wall clock; the last minute is for the assessment. | +| `--headed`, `--reporter`, `--output`, `--debug`, `--ai-trace`, `--trace [mode]`, `--video [mode]` | as `run` | Same meaning as for `e2e run`. The exploration is one attempt, so a retry mode (`on-first-retry`, `on-all-retries`) records nothing (the run's notice says to pass `--trace on`); put the goal before a bare `--trace` or `--video`. | + +Per-step action and model-call budgets default to 40 each; +`agents..maxSteps` and `agents..maxModelCalls` in the config +override them. The replay cache is off and retries are zero for the +exploration. + +## Reading the result + +Exit code `0`: steps ran, no `issue` was reported, and not every charter was +blocked; warnings are allowed. Exit code `1`: at least one `issue`, or no step +ran and nothing was found. If every charter was blocked and no `issue` was +reported, the run is `blocked` even with warnings, and the exit code is the +first blocker's own, as for `run`: 1 for an automation limit, 2 for a missing +model, credentials, seed data, or test setup, 3 for an unavailable +environment. When a charter passes, fails, or exhausts its budget, reported +issues decide the verdict. Other errors use `2` and `3` as for `run`. + +The terminal shows each step by its title with its duration, actions, and +findings, and each finding the moment it is reported as +`⚑ high issue Title (/path)`. At the end a `Findings` section lists every +finding, issues first and the most severe first, each with where it was seen, +its screenshot path, expected against actual, and the steps that reach it; +then the `Assessment` and the `run` summary with `Findings` and `Steps` in +place of `Test Files` and `Tests`. Severity words: critical 5, high 4, +medium 3, low 2, trivial 1. `.e2e/report.json` has the record under +`run.explore`: + +```json +{ + "goal": "...", + "budgets": { "maxSteps": 8, "timeoutMs": 600000 }, + "ended": "finished | step-limit | time | stuck | aborted", + "summary": "the closing assessment", + "steps": [{ "index": 1, "title": "...", "instruction": "...", "status": "passed | failed | blocked | exhausted", "summary": "...", "errorCode": "...", "startedAt": "...", "durationMs": 0 }], + "findings": [{ "id": "", "index": 0, "step": 1, "kind": "issue", "severity": 4, "title": "...", "expected": "...", "actual": "...", "reproduction": ["..."], "path": "/cart", "observationRevision": "...", "artifactId": ":artifact:2", "reportedAt": "..." }] +} +``` + +A step carries `errorCode` only when it did not pass; a finding carries +`step`, `path`, `observationRevision`, and `artifactId` only when known. +`artifactId` names the evidence screenshot among the attempt's `artifacts` in +the result whose `file` is `explore`, where its path, size, and digest are. +With `--session`, `run.results` also holds the setup's result and the +project's other tests as skipped (`filtered`). The attempt directory is +`.e2e/results/explore--/attempt-1/`: the slug is the goal's first +words in lowercase ASCII, capped, and the id keeps distinct goals, targets, and +agents apart while the same goal on the same target and agent always maps to +the same directory. For example +`.e2e/results/explore-check-the-cart-totals-1a2b3c4d5e6f7a8b/attempt-1/finding-1.png`. + +Turn a finding into a test: its `reproduction` steps are the `agent.act()` +instructions or `screen.*` actions, and `expected` is the assertion. + +## When it does not fit + +The agents entry's `tools` and `system` carry over to the explorer; a +hand-rolled `StepExecutor` under `executor` is replaced by the built-in agent +for the run, with a notice on stderr, and needs a `model` on the entry (or the +executor's) or explore fails before it starts. Findings are the model's claims +plus evidence, not verified reproductions: read `actual` against the +screenshot before filing a bug. diff --git a/adapters/e2e/vendor/e2e-skill/references/mcp.md b/adapters/e2e/vendor/e2e-skill/references/mcp.md new file mode 100644 index 0000000..1f52178 --- /dev/null +++ b/adapters/e2e/vendor/e2e-skill/references/mcp.md @@ -0,0 +1,134 @@ +# Driving the app over MCP + +`e2e mcp` serves a project's live app to a coding agent over MCP (stdio). The +coding agent drives the app the way the testing agent does: look at a screen +before writing a test, check a locator before committing to it. Running tests +and reading a failed run stay on the CLI (topics `running` and `debugging`). + +## Contents + +- Setup +- Tools +- Workflow +- Rules + +## Setup + +The server ships with `e2e`. `e2e init` offers to register it; by hand: + +```bash +claude mcp add e2e -- npx e2e mcp # Claude Code +``` + +Or declare it in the client's project config (`.mcp.json` for Claude Code, +`.cursor/mcp.json` for Cursor, `.vscode/mcp.json` for VS Code): + +```json +{ "mcpServers": { "e2e": { "command": "npx", "args": ["e2e", "mcp"] } } } +``` + +Sessions run on the project's config (`e2e.config.ts` or `e2e.config.mts`): +in a project without one, `open_session` fails with `CONFIG_NOT_FOUND`, so +set it up first (topic `setup`). No model is needed: without the optional `ai` package, sessions +open and the deterministic tools work. + +Flags: `--config ` names the default config file, `--target ` +fixes the target every session opens on, `--headed` shows the browser or +simulator when the engine supports it in sessions that do not set `headed` +(sessions are headless by default, as in `e2e run`), `--max-sessions ` +sets how many sessions may be open at once (default 4, 1 through 16). + +## Tools + +Four tools, all on the server registered as `e2e`; everything a session can +do is a catalog behind `call`. Clients show them with the server name, such +as `mcp__e2e__open_session` in Claude Code; when another MCP server offers a +tool with the same name, use the `e2e` server's. + +| Tool | Does | +| --- | --- | +| `open_session` | Loads the config (`config` names another file; default the nearest `e2e.config.ts`), starts the declared app command if any, boots the engine, opens the app URL, and returns the session id, the catalog, and the first observation. `target` is required when the config declares several. `headed: true` shows the browser or simulator when the user wants to watch; without it the session follows the server's `--headed`, headless by default. Each session has its own browser or device. | +| `tools` | The catalog: one line per tool with its argument names (`?` marks optional), the first sentence of its description, and `[read-only]` where it changes nothing. `tools {tool}` shows the full description and the JSON Schema of its arguments. | +| `call` | Runs one catalog tool: `call {tool: "tap", args: {target: "n42"}}`. Arguments are checked against the tool's schema first; a wrong one fails with `INVALID_ARGUMENT` naming the field. | +| `close_session` | Saves a recording still running, ends the attempt, disposes the engine, stops the app processes the session started once no other session uses them. | + +The catalog, per session: + +| Catalog tool | Does | +| --- | --- | +| `observe` | A fresh observation: one node per line as `#id role "name" ...`, plus the current path. On the web engine a link's `href` is its path on the app URL's origin, else origin and path (`mailto:` and `tel:` keep their scheme, `blob:` its inner origin and path); `?…` and `#…` mark a dropped query or fragment, `data:…` and `javascript:…` an inline payload, a trailing `…` a target cut at 256 characters. | +| `tap`, `double_tap`, `long_press`, `right_click`, `hover`, `type`, `press`, `select`, `check`, `scroll`, `scroll_to`, `drag`, `upload`, `navigate`, `back`, `dismiss_keyboard` | The grammar verbs, as the testing agent gets them: `check` takes `checked`, `drag` a `to` node, `upload` project-relative `files`; `dismiss_keyboard` comes with device engines. Each reports what changed on screen; `observe` shows the whole screen. A verb the engine cannot honor is not listed and fails with `UNSUPPORTED_CAPABILITY`. | +| `type_secret` | Fills a configured secret by name: a credential's password into a password field, a `secrets` entry into any editable input; the plaintext never reaches the agent. Listed when the config declares `credentials` or `secrets` and the engine can fill secrets. | +| `locate` | Tries a semantic locator with exactly one of `role`, `text`, `label`, `placeholder`, `testId` (`name` only with `role`; `exact` optional; anything else is `INVALID_ARGUMENT`) and returns how many nodes match, which, and the `screen.*` call to write. | +| `screenshot` | The masked pixels as an image. | +| `tap_at` | Taps a point (`x`, `y` in the latest screenshot's pixels): a listed control under it by id, otherwise the bare point, which an engine without a bare-point tap cannot do. Listed when the engine declares `tap` or a bare-point tap. | +| `hover_at` | Hovers a point the same way: a listed control under it by id, otherwise the bare point. Listed when the engine declares `hover` or a bare-point hover. | +| `type_at` | Types `value` into the field at a point: a listed input under it is filled by id; with a keyboard, anything else is tapped to focus it and typed into, at the caret unless `replace` is set. Without a keyboard a point on nothing listed fails. Listed when the engine declares `type` or a keyboard. | +| `press_at` | Sends one `key` (`Enter`, `Escape`, `Tab`) to the control at a point: a listed control gets it by id; with a keyboard, anything else is tapped to focus it and the key goes through the keyboard. Listed when the engine declares `press` or a keyboard. | +| `select_at` | Picks the option whose visible label is `value` in the select-like control at a point; the point must land on a listed select. Listed when the engine declares `select`. | +| `start_recording` | Starts a video of the app (`name` optional, for the file name). Listed when the engine records video. | +| `stop_recording` | Stops it and returns the absolute path of each video file, under `/videos//` (`.e2e` by default), or the URL of a provider's own recording. | +| Project tools | Every `defineTool` in the agent's `tools` that applies to the target's platform, under its own name; an engine pack such as `mobileTools` adds `open_app`, `swipe`, `alert`. | + +Project tools use their `toModelOutput` renderer for strings as well as +structured results. Its `error-text` and `error-json` outputs set the MCP +result's `isError: true`; treat them as failed calls. + +Once a secret is filled in the session, `screenshot` and the point tools +(`tap_at`, `hover_at`, `type_at`, `press_at`, `select_at`) answer with a +`PIXEL_TAINTED` line for the rest of it; act on listed nodes by id (topic +`writing-tests`). Before the session's first `screenshot`, a point tool only +says to take one. + +Resources: `e2e://guide` and `e2e://guide/` hold this skill. + +## Workflow + +1. `open_session`, then `call {tool: "observe"}` and act until the screen you + want to test is in front of you. Node ids are valid only for the newest + observation; an action reports what changed, so observe again before + using new ids. +2. `call {tool: "locate", args: {...}}` for each locator you intend to write. + One match: use the printed `screen.getByRole(...)` call. Zero or several: + adjust before writing the test; the same failure would hit the test as + `LOCATOR_NOT_FOUND` or `LOCATOR_AMBIGUOUS`. +3. Write `tests/.e2e.ts` (topic `writing-tests`). Deterministic steps + where you saw exact names; `agent.act` where the flow varies. +4. Run it from the shell: `npx e2e run tests/.e2e.ts`, read the + failure (topic `debugging`), fix, repeat. +5. `close_session` when you are done; an idle session closes on its own after + 30 minutes and never outlives 4 hours. When the client exits, every session + closes and the app commands stop. To look at another project or config, + `open_session {config: "path/to/e2e.config.ts"}`; no restart needed. + +## Rules + +- Record a demo or a bug for a pull request with `start_recording` once the + screen is set up, and `stop_recording` when the part worth watching is + over; `close_session` saves one still running. Videos are not masked: + keep secrets off screen while recording. +- A failed action is an error result that leads with the tool, its target, + and the code (`tap #n9 failed: LOCATOR_NOT_FOUND: ...`) and still shows + the screen it re-observed: re-aim from that screen. +- Nothing a session does is recorded as a test or into the replay cache. A + session is for looking and trying; the test is what you write afterwards. +- A run from the shell and a live session share the app only if the target's + app `command` sets `reuseExisting` (ignored in CI); otherwise close the + session before running. +- Parallel agents (subagents) share one server: each opens its own session + and passes its session id to every `tools`, `call`, and `close_session`. + A call may leave `session` out only while one session is open. Sessions + open at once share one config. Sessions on the same app command share its + process, which stops when the last of them closes. On mobile, two sessions + on one simulator fight over it: one target per device (topic `setup`), each + session on its own target. +- `TARGET_REQUIRED`: pass `target` to `open_session` or start with `--target`. + `UNKNOWN_TARGET`: no target of that name; the message lists the declared + ones. `NO_SESSION`: call `open_session` first, or the session named has + ended (the message says why). `SESSION_REQUIRED`: several sessions are open; + pass `session`. `SESSION_OPEN`: every session slot is taken (close one, or + raise `--max-sessions`). `CONFIG_IN_USE`: open sessions use another config; + open on theirs, or close them first. `ENGINE_IN_USE`: the target's engine + comes from a package every session shares; create it in the config or a + file it imports by path. `UNKNOWN_TOOL`: the name is not in this session's + catalog; the message lists what is. diff --git a/adapters/e2e/vendor/e2e-skill/references/running.md b/adapters/e2e/vendor/e2e-skill/references/running.md new file mode 100644 index 0000000..9aea05a --- /dev/null +++ b/adapters/e2e/vendor/e2e-skill/references/running.md @@ -0,0 +1,233 @@ +# Running tests + +## Contents + +- Commands +- The replay cache +- Output +- Exit codes +- Continuous integration + +## Commands + +```bash +npx e2e run [files...] [options] # run tests +npx e2e explore [goal] [options] # explore toward a goal without a test file (topic explore) +npx e2e list [files...] [options] # print what run would select +npx e2e init [directory] [--yes] # scaffold a project, refresh the agent skill +npx e2e guide [topic] # print this skill; topics: setup, writing-tests, agent, + # running, explore, debugging, mcp, bug-bash +npx e2e cache ls|clear|stats # inspect or empty the replay cache +npx e2e login|logout|models [provider] # e2e/oauth subscription logins: openai, + # github-copilot, opencode-console, spacexai +npx e2e mcp [--target ] # MCP server for a coding agent (topic mcp) +npx e2e feedback -m [opts] # report a problem with e2e itself +npx e2e telemetry [disable|enable] # anonymous usage telemetry: status or switch +``` + +`run` flags: + +| Flag | Effect | +| --- | --- | +| `[files...]` | Files, directories, quoted globs relative to the project root, or a bare name (`signup`, `signup.e2e.ts`, `agent/signup.e2e.ts` all select `tests/agent/signup.e2e.ts`); `file:line` is the test whose `test(` opens on that line. They narrow the config `tests` glob, never bypass it. | +| `--config ` | Config file; default `e2e.config.ts` or `.mts`, found upward. | +| `--target ` | Target names, comma-separated or repeated; only these start their app commands. Unknown names fail before startup. | +| `--tag ` | Any of the tags, comma-separated or repeated; all of them with `--tag-mode all`. An empty `--target`, `--tag`, or `--agent` value is a usage error, exit 2. | +| `--exclude-tag ` | Drop tests carrying any of these tags, however selected. | +| `--grep `, `--grep-invert ` | Keep, or drop, tests whose title (describe titles and test title joined by spaces, `checkout pays`; not file or tags) matches a regular expression. Bare pattern or `'/pattern/i'`; repeat for alternatives. | +| `--last-failed` | The tests the previous run (`/report.json`) did not pass or never ran (`--max-failures` included), plus every test in a failed `beforeAll` or `afterAll` scope. A test another filter leaves out stays owed (`run.carried` in the report) until a rerun runs it. No report is `NO_LAST_RUN`, exit 2. | +| `--shard ` | One contiguous slice of the selected tests (`--shard 2/3`), cut after every other filter; serial groups stay whole, each shard brings its own setup tests. | +| `--headed` | Visible browser or simulator when the engine supports it. | +| `--agent ` | Run unpinned tests as these `agents.` entries (default `agents.default`), comma-separated or repeated; several names run each such test once per agent. | +| `--workers `, `--retries ` | Override the resolved values; retries 0-10, workers 1-1024. | +| `--max-failures ` | Stop after n failures: the rest skip (cause `failure-limit`), running tests end `interrupted`; exit 1. | +| `--repeat-each ` | Run every selected test n times, each run its own result (`repeat` 0 through n-1); add `--no-cache` or later runs replay the first's recording. The `Repeats` summary row names each unstable test's failed runs. | +| `--reporter ` | `list`, `json`, `junit`, `markdown`, comma-separated; `json` cannot combine with `list`. | +| `--output ` | Results directory, over the config's `output` (default `.e2e`). | +| `--no-cache` | Replay cache off for this run. | +| `--strict-cache` | Fail a step whose committed recording no longer replays (`REPLAY_STALE`, exit 2) instead of handing it to the agent; retries replay too, and the run never writes the cache. | +| `-u`, `--update-snapshots` | Write the stored screenshots `toHaveScreenshot` finds missing or different, and pass. | +| `-w`, `--watch` | Run, then run again when a test file, a source file beside the tests, or the config is saved, or a key asks: `a` or enter all, `r` the current filters, `f` failed, `u` update snapshots, `p` and `t` filter by file and title, `q` quit; a key during a run cancels it. Refused in CI and with `--reporter json`. | +| `--pass-with-no-tests` | Exit 0, not `NO_TESTS`, when nothing matches. | +| `--debug` | Phase timings and an agent step table on stderr; transcripts as artifacts. | +| `--ai-trace` | Every model call, to `/ai-trace.json`. | +| `--trace [mode]`, `--video [mode]` | Which tests keep a trace page (`/results//trace.md`), or which attempts record a video (WebM on browsers, MP4 on devices), over the config and every target: bare is `on` (`--trace` then pages every test, passing ones too); `--trace off` writes none; `retain-on-failure` keeps only failures; `on-first-retry` keeps first retries, `on-all-retries` every retry. A test's own `trace` or `video` still wins; targets whose engine cannot record video are skipped with a notice. Both are greedy: write `--video=` or put test files first. The failure recap names the trace page and the video. | + +```bash +npx e2e run tests/signup.e2e.ts +npx e2e run signup.e2e.ts:12 # by bare name, the test declared at line 12 +npx e2e run tests/agent --tag smoke --exclude-tag slow --grep checkout +npx e2e run --last-failed # the loop after a red run +npx e2e run --shard 2/3 # one CI job of three +npx e2e run 'tests/**/*.smoke.e2e.ts' --target chromium --workers 1 --retries 0 +CI=1 npx e2e run # the CI defaults, locally +``` + +`list` takes the same files and selection flags (`--config`, `--target`, +`--tag`, `--tag-mode`, `--exclude-tag`, `--grep`, `--grep-invert`, +`--last-failed`, `--shard`, `--pass-with-no-tests`), prints one line per +test-target pair, `file › title [target] #tag`, skipped pairs ending in +` (skipped: )`, and starts no app, engine, or worker. +`--reporter json` prints `{ "pairs": [...] }` and names a skip reason `reason`. + +`import { list } from 'e2e/runner'` returns that selection plus the pairs a +filter removed, positional arguments that matched no file, and the selected +target names. + +```bash +npx e2e list tests/signup.e2e.ts --tag smoke --reporter json +``` + +With a `package.json` script `"test:e2e": "e2e run"`, pnpm forwards `--` +literally: `pnpm test:e2e -- --headed` reaches e2e as `run -- --headed` and +exits 2. Write `pnpm test:e2e --headed` or `pnpm exec e2e run --headed`. +With npm, use `npm exec -- e2e run --headed` to pass flags to e2e. + +## The replay cache + +Entries live under `.e2e/cache/`, one file per key, named by key digest. +`cache` commands read the same config as `run` (`--config`, `cache.dir`); +with a custom `cache.store` they refuse (exit 2), inspect that store with +its own tools. + +| Command | Prints | +| --- | --- | +| `e2e cache ls` | One row per entry: test, target, instruction digest, age, action count. | +| `e2e cache stats` | Directory, entry count, total size. | +| `e2e cache clear` | Deletes the entries and the directory; files the runner never wrote stay. | + +## Output + +`` (`.e2e` by default) holds `report.json`, `junit.xml`, +`summary.md`, `ai-trace.json`, `sessions/`, and `results/`, one directory +per test with its `trace.md` and an `attempt-/` per attempt +(screenshots, videos, `--debug` transcripts, downloads). +`results/` is cleared once a run's tests start; a run stopping before +leaves the last run's files, and a `--last-failed` rerun keeps the files +the report it reruns names and writes its own under each test's `rerun-/`. The report records every artifact path, a +hosted service's video by URL. + +- `list` (default): setup steps, one line per file and target, a `Failed + Tests` section (error, code, failing line, code frame), then the + summary rows `Test Files`, `Tests`, `AI`, `Cache` (when the replay cache + ran), `Repeats` (with `--repeat-each`), `Errors`, `Start at`, `Duration`, + `Traces` (when a page was written), `AI trace` (with `--ai-trace`). Past a minute `Duration` + repeats as minutes and seconds, setup time split out as `startup` in the + same parenthetical (`682.97s (11m 23s, startup 43.00s)`). +- `report.json`, for scripts rather than reading (`jq`, CI tools; a reader + takes the terminal and the trace pages), the one record of every result, + passing ones included, written whatever the reporters, holds `run.status`, + `run.exitCode`, `run.errors[]` (run-level, such as `APP_UNREACHABLE`), + and `run.results[]`, one per test and target: `titlePath`, `file`, + `source`, `tags` (`[]` when none), `agent`, `repeat` (0 unless + `--repeat-each`), `selected`, `status`, `attempts[]` of `steps[]`, + `artifacts[]`, `error`. `run.summary` counts the selected results: + `passed`, `failed` (timed out included), `interrupted` (stopped by the + run: a signal, `--max-failures`, a run-level error; not a failure), `flaky`, and `skipped` add up + to `selected`; `discovered - selected` were left out. +- `junit`: `junit.xml` for CI summaries; `--reporter list,junit` keeps the + terminal output. +- Every run that reaches its tests also writes a trace page per failed, + timed-out, interrupted, or flaky test, `results//trace.md` (the + `trace` mode, default `retain-on-failure`): every step with what + it did, the cache's decisions, what the app logged, the last model turns, + and the screen at failure. The `list` output names it under each failure + (`❯ trace `). Read it first. +- `markdown` (`--reporter list,markdown`): `summary.md` with counts and + spend, a block per failed test linked to its page, the flaky tests folded + alike, and every test as one folded table (or an exploration's findings + and assessment). Paste it into a pull request or handoff. +- `json`: the report on stdout. +- Custom reporters get step progress with `identity` (`attemptId`, + `attemptIndex`, `stepId`, `stepIndex`: report IDs, zero-based indexes; + retries change the attempt, serial members share the group's attempt with + distinct step IDs) and an `end` phase carrying the redacted error and the + `blocked` and `cancelled` statuses. Older streams may lack `identity`. +- `github()` from `@e2e-dev/github`: on GitHub Actions, one pull request + comment per run (edited on rerun) plus the job summary; needs + `pull-requests: write` and `GITHUB_TOKEN` (or `GH_TOKEN`) in the step's + env. + +## Exit codes + +| Code | Meaning | +| ---: | --- | +| 0 | Every selected test passed, was flaky, or was skipped | +| 1 | A test or setup test failed or timed out | +| 2 | CLI, config, collection, credential, model-config, or policy error | +| 3 | Engine, app process, model provider, artifact, or cleanup failure | +| 4 | Internal runner error | +| 130 | Interrupted by an external signal | + +The highest code present wins (`130 > 4 > 3 > 2 > 1 > 0`). 130 needs an +external signal: a `--max-failures` stop or a run-level error that +interrupted workers keeps the failures' or the error's code. Exit 2 is +deterministic, never retry it; only exit 3 is worth a job-level retry. + +The first SIGINT or SIGTERM interrupts and writes the report if any test +had started; the second forces engine teardown and still writes the +report, plus `junit.xml` and `summary.md` when those reporters are on; the +third kills the app process groups and exits 130 at once. + +## Continuous integration + +CI mode is on when `CI` is set (not `0` or `false`): `retries` 1, +`workers` 1, `test.only` rejected with +`ONLY_IN_CI`, the replay cache `read-only` unless the config sets a mode explicitly +(`cache: 'read-write'` or `cache.mode`), `reuseExisting` ignored. + +```yaml +# .github/workflows/e2e.yml +# Omits the github() reporter: it needs pull-requests: write and +# GITHUB_TOKEN in the run step's env. +name: e2e +on: + pull_request: + push: + branches: [main] +permissions: + contents: read +jobs: + e2e: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: pnpm/action-setup@9fd676a19091d4595eefd76e4bd31c97133911f1 # v4.2.0 + - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + with: + node-version: 26 # Node >= 24.8, or >= 22.22.3 on 22 + cache: pnpm + - run: pnpm install --frozen-lockfile + - run: pnpm exec e2e-web install chromium --with-deps + - run: npx e2e run --reporter list,junit + env: + AI_GATEWAY_API_KEY: ${{ secrets.AI_GATEWAY_API_KEY }} + E2E_USER_ADMIN_USERNAME: ${{ secrets.E2E_USER_ADMIN_USERNAME }} + E2E_USER_ADMIN_PASSWORD: ${{ secrets.E2E_USER_ADMIN_PASSWORD }} + - if: ${{ !cancelled() }} + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: e2e-report + path: | + .e2e/report.json + .e2e/junit.xml + if-no-files-found: warn + - if: ${{ !cancelled() }} + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: e2e-artifacts + path: .e2e/results + if-no-files-found: warn + retention-days: 7 +``` + +- Install browsers as their own step so the download stays out of the + launch timeout. +- Upload artifacts unless cancelled, so a test that failed then passed on + retry keeps its evidence. +- Start the app through the target's `app.command`; the runner tears it down + on every exit path. +- Agent steps run in the same job: pass the key the config's model reads + (`AI_GATEWAY_API_KEY` for `gateway()` from `ai`) as a secret in the run + step's `env`, and commit `.e2e/cache/` so recorded steps replay with no + model call. diff --git a/adapters/e2e/vendor/e2e-skill/references/setup.md b/adapters/e2e/vendor/e2e-skill/references/setup.md new file mode 100644 index 0000000..07b2687 --- /dev/null +++ b/adapters/e2e/vendor/e2e-skill/references/setup.md @@ -0,0 +1,390 @@ +# Setting up e2e + +## Contents + +- Requirements +- Scaffold +- Subscriptions and API keys +- The config +- The app under test +- Environment variables the runner reads +- Mobile targets +- Done when + +## Requirements + +- Node.js 24.8 or newer, or 22.22.3 or newer on Node.js 22. +- ES modules: `.ts` config, tests, helpers, and workspace packages exporting + `.ts` source load as ESM regardless of the nearest `package.json` `type` + (CommonJS packages need no change); never `require` or `module.exports`. + Imports follow TypeScript: `./x.js` or `./x` loads `x.ts`, and the nearest + `tsconfig.json` `paths` and `baseUrl` apply. Decorators need + `experimentalDecorators`; CommonJS TypeScript goes in `.cts`. +- Browser tests: `@e2e-dev/web`, pinning `playwright-core` exactly; do not + add `playwright` for it. Missing browsers download on first boot; in CI run + `npx @e2e-dev/web install chromium --with-deps` (pnpm: `pnpm exec e2e-web + install chromium --with-deps`). Mobile tests: `@e2e-dev/mobile`, pinning + `agent-device` exactly; each pin moves with its engine release. + +## Scaffold + +Fresh project: + +```bash +npx e2e init # npm +pnpm dlx e2e init # pnpm +``` + +With `e2e` installed, run the installed version: `npx e2e init` or `pnpm exec +e2e init`; `npx e2e init my-app` scaffolds into a new directory. + +The wizard picks an engine and a model provider (None for tests without AI) +and offers to install this skill, register the MCP server for your coding +agent, and install dependencies. `--yes` picks Playwright and Vercel AI +Gateway, installs the skill in `.agents/skills/` (`.claude/skills/e2e` +symlinks to it), registers MCP in `.mcp.json` and `.cursor/mcp.json`, skips +installing dependencies. + +Init adds dependencies and a `test:e2e` script to `package.json`, writes +`e2e.config.ts` and `tests/example.e2e.ts`, updates `.gitignore`, leaves +existing configs and tests alone. Re-run after upgrading to refresh skill and +MCP entries. + +Without the wizard (`ai`, Vercel AI SDK v7, only for `agent.*` steps): + +```bash +npm install --save-dev e2e @e2e-dev/web ai@^7 +``` + +## Subscriptions and API keys + +`e2e init` writes model config and dependencies for a subscription, an API +key, or a local endpoint. Authenticate: + +| Choice | Setup | +| --- | --- | +| ChatGPT Plus or Pro | `npx e2e login openai` | +| GitHub Copilot | `npx e2e login github-copilot` (GitHub CLI signed in, or your own `--client-id`) | +| OpenCode Console (OpenCode Zen and OpenCode Go) | `npx e2e login opencode-console` (approve the device code, pick the workspace) | +| SuperGrok or X Premium+ | `npx e2e login spacexai` | +| Claude Max or Team | No `e2e login`: link a Claude Console organization in claude.ai for the plan's monthly API credits (Max: Settings > Billing; Team: an Owner under Organization settings > Billing), create an API key in it, set `ANTHROPIC_API_KEY`, use `anthropic('')` from `@ai-sdk/anthropic` | +| Vercel AI Gateway | Set `AI_GATEWAY_API_KEY`, or sign in to the Vercel CLI and `npx vercel link`; without the key `gateway()` uses a Vercel OIDC token | +| OpenRouter | Set `OPENROUTER_API_KEY` | +| Local or self-hosted endpoint | Set the endpoint URL and a model it serves, plus a key if required | + +Switching an existing config to ChatGPT: install `ai` and `@ai-sdk/openai`, +set `model: chatgpt('gpt-6-luna')` from `e2e/oauth/chatgpt`, run `npx e2e +login openai`. `npx e2e models` lists the ids each login serves. Use API keys +in CI. + +Switching to Copilot: install `ai`, `@ai-sdk/openai-compatible`, and +`@ai-sdk/openai`, set `model: copilot('')` from `e2e/oauth/copilot`, run +`npx e2e login github-copilot`. `copilot()` calls a model over chat completions, or +over Copilot's Responses API when the plan serves that model only there, choosing +per model from the plan's listing. `npx e2e models github-copilot` marks the models +it cannot call at all: those served only over an API `copilot()` does not speak, and +those the plan has not enabled. + +Switching to OpenCode Console: install `ai`, `@ai-sdk/openai-compatible`, +`@ai-sdk/openai`, `@ai-sdk/anthropic`, and `@ai-sdk/google`, set +`model: opencodeConsole('')` from `e2e/oauth/opencode-console`, run +`npx e2e login opencode-console`. A bare id is an OpenCode Zen model; a `go/` id +(`go/deepseek-v4.1-flash`) is an OpenCode Go model and needs the workspace's Go +subscription. `npx e2e models opencode-console` lists the ids, tagged Zen or Go. In CI, +set a Console service account key as `OPENCODE_API_KEY`. + +## The config + +`e2e.config.ts` sits at the project root and default-exports an object literal +ending in `satisfies E2EConfig`; unknown keys are `INVALID_CONFIG` at load. + +```ts +import type { E2EConfig } from 'e2e'; +import { web } from '@e2e-dev/web'; +import { gateway } from 'ai'; + +export default { + tests: 'tests/**/*.e2e.ts', + targets: [ + { + engine: web(), + app: { + url: 'http://127.0.0.1:3000', + command: { executable: 'pnpm', args: ['dev'], log: '.e2e/logs/app.log' }, + }, + }, + ], + // Model behind every agent.* step. + agents: { + default: { + model: gateway('openai/gpt-6-luna-fast'), + system: 'You are a thorough QA agent. Verify every outcome on screen.', + }, + }, + credentials: { + admin: { username: 'admin@example.test', password: process.env.ADMIN_PASSWORD ?? '' }, + }, +} satisfies E2EConfig; +``` + +A string `password` is checked at load (6+ code points): set `ADMIN_PASSWORD` +or `E2E_USER_ADMIN_PASSWORD` first, or defer to fill time with +`() => process.env.ADMIN_PASSWORD ?? ''`. + +| Key | Default | Notes | +| --- | --- | --- | +| `targets` | required | Non-empty; `--target` takes `name`. UI targets set `engine` and `app` (`platform` and `name` default to the engine's platform); tools-only targets may omit `engine` and must set `platform`. | +| `tests` | `'tests/**/*.e2e.ts'` | Globs relative to the project root (optional `./`): `*`, `?`, whole `**` segments, leading `!` excludes (`'!tests/wip/**'`; only exclusions is `INVALID_CONFIG`). Braces, character classes, extglobs, `..`, absolute paths, and a wildcard-free directory entry (`'!tests/wip'`) are `INVALID_GLOB`. | +| `timeout` | `120000` | Per test attempt, ms; also the default `agent.act` deadline. | +| `launchTimeout` | `60000` | Engine init and attempt start, ms. | +| `actionTimeout` | `30000` | Each locator action and engine operation, agent-step observations included. | +| `assertionTimeout` | `5000` | `expect` polling window. | +| `cleanupTimeout` | `30000` | Each `afterEach` hook, fixture teardown, engine cleanup, ms. | +| `retries` | `0`, `1` in CI | 0 to 10. | +| `workers` | half the cores, `1` in CI | Parallel test files, capped by the `workers` the engine declares per target. | +| `reporters` | `['list']` | `list`, `json`, `junit`, `markdown`, or `{ name, onEvent?, onRunFinished? }` objects; `json` excludes `list`, `--reporter` keeps the objects. | +| `cache` | `'read-write'` | `'read-write'`, `'read-only'`, `'off'`, or `{ mode, store, dir, strict }`; CI demotes only a defaulted mode to `'read-only'`. `strict` (`--strict-cache`) fails a step whose recording no longer replays (`REPLAY_STALE`) instead of handing it to the agent; retries replay too, and the run never writes the cache. | +| `agents` | `{ default: built-in }` | Tests run with `default`, `e2e run --agent ` picks another, entries never inherit from `default`. Options: topic `agent`. | +| `credentials` | `{}` | Named `{ username, password }`; `password` is a 6+ code point string or a function returning it. | +| `secrets` | `{}` | Named values the model never sees (API keys, tokens), same value rule. A separate namespace: a credential's password is `credentials.user(name).password` (named `.password`), never `secrets.get()`, so a name may be both. | +| `output` | `'.e2e'` | Results directory; `--output ` for one run. Inside the project root, not the root, not a tests glob's directory, never the cache dir (`cache.dir` stays `.e2e/cache`). | +| `artifacts` | none | `{ store }`: artifacts go to the host `ArtifactStore` (`{ put(artifact), putLink?(link) }`); `putLink` gets provider-hosted video links (never a passed `retain-on-failure` attempt's). Failure screenshots are always captured when the engine can. | +| `trace` | `'retain-on-failure'` | Tests that keep a trace page (`/results//trace.md`): `'off'`, `'on'` (every test), `'retain-on-failure'` (failed, timed-out, flaky), `'on-first-retry'`, `'on-all-retries'`; precedence as `video`. Works on every engine. | +| `video` | `'off'` | Attempts that record, same modes; `'retain-on-failure'` records all, keeps those that did not pass. Precedence: the test's `video`, `--video [mode]`, the target's (`{ engine, video }`), the config's. | +| `projectId` | the package name | Report and cache identity. | + +- `tests` discovery enters only directories a glob can match; symlinks are + not followed. +- `video`: a config or flag mode skips targets whose engine cannot record + (one notice), a target or test mode requires it (`UNSUPPORTED_ARTIFACT`). +- `trace` and `video`: a retry mode with `retries: 0` prints a notice; + neither invalidates the replay cache. + +## The app under test + +The target declares the app; the engine only drives it. `web({ url })`, +`mobile({ app })`, and the other old app options are unknown keys. The +target's `app`: + +| Key | Meaning | +| --- | --- | +| `url` | Base URL for `app.open()` and relative navigation; required on a `web()` target, not supported on a mobile target yet. Missing scheme: `https://`, `http://` for loopback; port `0` on `127.0.0.1` or `[::1]` takes a free port. | +| `bundleId` | Device targets: the bundle id, package name, or display name (`Settings`) `app.open()` launches. | +| `appPath` | Device targets: the `.app` or `.apk` under test. A device target needs `bundleId` or `appPath`. | +| `launchArguments`, `permissions` | Device targets: arguments and permission states (`grant`, `deny`, `reset`) every fresh launch gets. | +| `command` | The process serving `url`: `{ executable, args, cwd, env, startupTimeout, shutdownTimeout, log, reuseExisting }`; `{port}` in `args` and `env` expands to `url`'s port. Targets declaring the same command share one process. | +| `readyUrl` | Readiness probe when it differs from `url`; `{port}` expands too. | +| `environment` | `'test'`, `'staging'`, `'production'`; inferred from the host, labels the report and cache key. | +| `identity` | Stable identity for cache and session keys when the origin changes per deploy (preview URLs). Defaults to the URL's origin and path, else `bundleId`, else `appPath`. | + +A Next.js 16 dev server blocks cross-origin requests to its dev resources, so +a target that opens `127.0.0.1` or `[::1]` while `next dev` identifies as +`localhost` never hydrates. Add the host to `allowedDevOrigins` in +`next.config.ts` (`'127.0.0.1'`, or `'[::1]'` for the bracket form). A +`localhost` target needs no entry, and a production `next start` target is +unaffected. + +There is no `services` key in this version: it is an unknown key wherever it +appears. Start dependency processes before the run, or have `app.command` +start a script that brings them up and serves the app. + +`web()` options: + +| Option | Meaning | +| --- | --- | +| `browser` | `'chromium'` (default), `'firefox'`, `'webkit'`, or a `BrowserProvider` leasing hosted browsers over CDP (`kernel()` from `@e2e-dev/kernel`, `smol()` from `@e2e-dev/smol`, or your own), which implies chromium and excludes `connect`. Scope `'worker'` (default): one browser per worker slot from `prepare` to `finish`; `'attempt'`: one per attempt, with `reconnectEndpoint`'s limits. | +| `viewport` | `{ width, height }`, default 1280x720; `null` follows the browser window (hosted live view, headed run). On a headed hosted browser (Kernel) use `null` and size the service's screen; a fixed size gives a smaller, unmaximized window. | +| `connect` | `{ cdpEndpoint }` attaches to a remote Chromium over CDP; both it and `reconnectEndpoint` are resolvers `(signal) => url`, not strings. With `reconnectEndpoint` it rides one persistent default context and reconnects only to the original browser and page. | +| `headers` | Sent to the app's site only (Vercel's `x-vercel-protection-bypass`, ngrok's `ngrok-skip-browser-warning`), `agent.act` included; disables the browser HTTP cache and service workers. | +| `basicAuth` | `{ username, password }` for a `401` challenge; `password` may be `secrets.get('name')`, resolved per attempt and redacted like any secret, the base64 `Authorization` credential too. | +| `userAgent` | The `User-Agent` every attempt sends and `navigator.userAgent` reports. | +| `locale`, `timezoneId` | The language (`'de-DE'`: `navigator.language`, `Intl`, `Accept-Language`) and IANA time zone (`'Europe/Berlin'`) every attempt runs in. | +| `initScripts` | Scripts every document runs before the page's own: source, `{ path }`, or a function with no closures. | +| `testIdAttribute` | What `getByTestId` reads; default `data-testid`. | +| `screencast` | `{ size?, quality? }` for the engine's own video: frame size (default the viewport's), JPEG quality 0 to 100. | + +- `basicAuth` via `secrets.get()` masks text, not screenshots or model pixels + showing the password. An undeclared name is `INVALID_CONFIG` at load, as is + the handle (a reference, not the value) in `app.command.env`, a template + literal, or `context`; read those from `process.env`. +- `reconnectEndpoint` or an attempt-scoped provider rides one persistent + context without `headers`, `basicAuth`, `userAgent`, `locale`, `timezoneId`, `app.clearState()`, or + session state. Recovery never repeats a dispatched operation; exhausting + the budget is `OPERATION_TIMEOUT`. Without `reconnectEndpoint` a dropped + connection is reacquired at the next attempt. + +Two browsers, one app declaration: + +```ts +const app = { url: 'http://127.0.0.1:3000' }; +export default { + targets: [ + { name: 'chromium', engine: web(), app }, + { name: 'mobile-webkit', engine: web({ browser: 'webkit', viewport: { width: 390, height: 844 } }), app }, + ], +} satisfies E2EConfig; +``` + +### Let the runner start the app + +Prefer `app.command` to a hand-started dev server: self-contained locally +and in CI. + +```ts +engine: web(), +app: { + url: 'http://127.0.0.1:3000', + command: { + executable: 'pnpm', + args: ['dev'], + env: { PORT: '3000', DATABASE_URL: process.env.DATABASE_URL ?? '' }, + startupTimeout: 120_000, + log: '.e2e/logs/app.log', + }, +}, +``` + +- The runner spawns `command`, polls `readyUrl` (default `url`) for a 200 to + 499 status within `startupTimeout` (default 60 s), and stops it when the run + ends, fails, or is interrupted (`shutdownTimeout`, default 10 s). Never + ready is `APP_UNREACHABLE`; the run stops before its tests and leaves the + last run's report and results as they were (`command.log` still gets this + boot's output). +- The child gets only `PATH`, `HOME`, the temp-directory variables, + `SystemRoot` and `COMSPEC` on Windows, and `command.env`; pass the rest + through `env`. Model keys and `E2E_USER_*` values are never inherited. +- Output is discarded unless `log` names a file under an ignored directory + such as `.e2e/logs/`; without it a server dying on boot is invisible. +- A `url` answering before the spawn is `APP_ALREADY_RUNNING`; + `command.reuseExisting: true` attaches to a running dev server (CI ignores + it). +- `executable` resolves on `PATH`, never through a shell; name the server + itself, not a wrapper script. +- `url: 'http://127.0.0.1:0'` (or `[::1]:0`, never `localhost:0`) takes a + free port; the command receives it as + `{port}` in `args` or `env` (`args: ['dev', '--port', '{port}']`), which + also expands in `readyUrl`. Tests read the URL from `app.baseUrl`; the + cache identity keeps `:0`. A port-0 `url` with no `command`, and + `reuseExisting` beside one, are `INVALID_CONFIG`. + A port grabbed between allocation and spawn fails the start with + `APP_UNREACHABLE`; rerun. + +For an app started elsewhere, point `app.url` at it, literally or via +`process.env.APP_URL ?? 'http://localhost:3000'`; the runner reads no +`APP_URL` and loads no `.env`, so put `process.loadEnvFile('.env')` atop +`e2e.config.ts` (workers re-import it). + +## Environment variables the runner reads + +| Variable | Effect | +| --- | --- | +| `AI_GATEWAY_API_KEY`, `OPENROUTER_API_KEY`, `OPENAI_API_KEY`, ... | Read by provider packages, not the runner. | +| `E2E_USER__USERNAME`, `E2E_USER__PASSWORD` | Override `credentials.`; `` is the name uppercased, other characters `_`. | +| `E2E_SECRET_` | Overrides `secrets.`, same rule. Two entries of one namespace mapping to one variable are `INVALID_CONFIG`. | +| `CI` | CI defaults; list in topic `running`. | +| `E2E_TELEMETRY_DISABLED`, `DO_NOT_TRACK` | Disable anonymous telemetry, as does `e2e telemetry disable`; `E2E_TELEMETRY_DEBUG=1` prints events instead of sending. | + +## Mobile targets + +`@e2e-dev/mobile` drives iOS simulators, Android emulators, and connected +phones through [agent-device](https://github.com/callstack/agent-device); +needs Xcode with a simulator runtime or the Android SDK with an emulator (a +phone needs Xcode with Developer Mode and runner signing, or adb with USB +debugging authorized); run `npx agent-device doctor` once. Name a phone in `device` by the name +`npx agent-device devices` lists, not its UDID or serial. On an iPhone, device +settings (permissions, `clearState`, network, location, appearance, +biometrics, keychain) and the clipboard are simulator-only. + +```ts +import type { E2EConfig } from 'e2e'; +import { mobile } from '@e2e-dev/mobile'; +import { mobileTools } from '@e2e-dev/mobile/tools'; +import { gateway } from 'ai'; + +const iphone = mobile({ platform: 'ios' }); + +export default { + targets: [{ engine: iphone, app: { bundleId: 'com.example.app' } }], + workers: 1, + agents: { + default: { + model: gateway('openai/gpt-6-luna-fast'), + tools: mobileTools(iphone), + }, + }, +} satisfies E2EConfig; +``` + +- `app.bundleId`: bundle id, package name, or display name `app.open()` + launches fresh (an attempt launches nothing itself; without it, the + installed build). `app.appPath`: the `.app` or `.apk` under test; the + engine installs nothing, so a fixture every test takes calls + `device.installApp()` once per device (no path installs `app.appPath`). + `app.launchArguments` and `app.permissions` ride every fresh launch: + arguments reach the app process (iOS) or `am start` (Android), permissions + are set first. +- Expo development build on an iOS simulator: `app.launchArguments: + ['--initialUrl', 'http://localhost:8081', '-EXDevMenuShowsAtLaunch', 'NO', + '-EXDevMenuIsOnboardingFinished', 'YES', '-EXDevMenuShowFloatingActionButton', + 'NO']` loads that dev server on every fresh launch, dev menu off; + `app.command` (`npx expo start --port 8081`) with `readyUrl: + 'http://localhost:8081/status'` starts it. +- One worker per device. No `device`: every booted simulator or emulator of + the platform, and every connected phone, is the pool, up to `workers` (none booted: agent-device boots + one); one `device`: one worker whatever `workers` says; a list (`device: + ['iPhone 17', 'iPhone 17 Pro']`): an explicit pool. Devices boot in + `prepare`, before the run's clock. Two sessions on one device fight over + it: for parallel MCP sessions or bug-bash explorers, declare one target per + device, each naming its `device`. +- `device` can be a `DeviceProvider` leasing hosted devices, one per worker + slot: `easSimulators({ buildId })` from `@e2e-dev/eas` takes the project + from the app config's `extra.eas.projectId` (`projectId` overrides), reads + `EXPO_TOKEN`, else the `eas login` session, needs no Xcode or Android SDK, and with `buildId` EAS installs + the app (omit `app.appPath`). A run must fit one session: `maxDurationMinutes`, + absent, is the account's cap (40 on a standard plan). `videoTouches: false` + on the engine for video there. +- `limrun()` from `@e2e-dev/limrun` leases one device per worker and installs + `app.appPath` before tests. Set `LIMRUN_API_KEY`; install `@limrun/api` and + matching `agent-device` peers. Android needs `adb` on `PATH` for its tunnel. + With agent-device 0.21.22, iOS keyboard actions such as `locator.press('Enter')` + return `UNSUPPORTED_CAPABILITY`; tap an app button when available. + See `examples/with-limrun` for the same form and restart tests on both platforms. +- Only a control that appeared or moved with the previous action waits out + `transition` (default 500 ms); agent actions settle `settle` ms (default + 150) before the next observation, `settle: false` skips it. +- `screen`, `expect`, `app`, `agent` work unchanged; `test` from + `@e2e-dev/mobile` types the `device` fixture (`installApp`, `openLink`, + `setPermission`, `setNetwork`, `setAppearance`, `clearKeychain`, `fold` + for a foldable's hinge (iPhone Duo, Android foldable emulators), `locator`, more). Portable suites declare + `requires: ['device']`. +- `setNetwork('offline')` and `setAirplaneMode(true)` are Android only: on + iOS both are `UNSUPPORTED_CAPABILITY`, since a simulator shares the Mac's + network, and `setNetwork('online')` / `setAirplaneMode(false)` do nothing. + `setNetwork('offline')` turns off Wi-Fi alone and cellular stays up; an + offline test uses `setAirplaneMode(true)` (Android 11+) under + `platforms: ['android']` and turns it back off in `afterEach`. +- No `state` capability: `test.setup` and `session` are unavailable; sign in + per test with `screen` actions or `agent.act`, both fill a `Secret` (topic + `writing-tests`, Sign-in sessions). +- A deterministic check naming a platform label runs there only: + `test('...', { platforms: ['ios'] }, ...)`. +- `selectOption`, `setInputFiles`, `scrollIntoView`, `secondaryTap`, and + `modifiers` on `tap` or `doubleTap` are `UNSUPPORTED_CAPABILITY` on a + device. +- React Native on iOS: checkbox and radio role and state come off the + accessibility value (`getByRole`, `check()`, `toBeChecked` work); a tab is + `other` with `selected`, query by test id or label; a plain `View` is a + leaf, scope to the `ScrollView` or give children test ids. + +## Done when + +- `npx e2e run tests/example.e2e.ts` passes against the app. +- `package.json` has a script such as `"test:e2e": "e2e run"`. +- `.gitignore` lists the `.e2e/` outputs (init adds them); drop the + `.e2e/cache/` line to commit replays. +- CI runs the whole suite, agent steps included, on PRs; see `running`. + +### Existing MCP registration + +Re-running `init` preserves the e2e MCP server's environment and client settings +while repairing its launch command. A current registration is left unchanged. diff --git a/adapters/e2e/vendor/e2e-skill/references/writing-tests.md b/adapters/e2e/vendor/e2e-skill/references/writing-tests.md new file mode 100644 index 0000000..8a38a80 --- /dev/null +++ b/adapters/e2e/vendor/e2e-skill/references/writing-tests.md @@ -0,0 +1,443 @@ +# Writing tests + +## Contents + +- A complete file +- Registration +- Fixtures +- Locators +- expect +- Sign-in sessions +- The browser fixture (browser only) +- Habits + +## A complete file + +```ts +// tests/todos.e2e.ts +import { beforeEach, describe, test } from '@e2e-dev/web'; +import { expect } from 'e2e'; + +describe('todos', { tags: ['todos'] }, () => { + beforeEach(async ({ app }) => { + await app.open('/todos'); + }); + + test('adds and completes a todo', async ({ agent, screen, browser }) => { + await agent.act('add a todo named {title}', { params: { title: 'Write the release notes' } }); + await expect(screen.getByRole('listitem')).toHaveCount(1); + await expect(screen.getByRole('status', 'Remaining')).toHaveText('1 remaining'); + + await agent.act('mark the todo as done'); + await expect(screen.getByRole('status', 'Remaining')).toHaveText('0 remaining'); + await expect(browser).toHaveURL('/todos'); + }); + + test('ignores an empty submission', async ({ screen }) => { + // An exact interaction: the empty submit is the point of the test. + await screen.getByRole('button', 'Add').tap(); + await expect(screen.getByRole('listitem')).toHaveCount(0); + }); +}); +``` + +The agent does the flow; `expect` pins what must be true after each goal, +and that check lets the replay cache rerun the step later. `screen` actions +are for exact interactions and values. Files match the config `tests` glob, +default `tests/**/*.e2e.ts`. In a browser every test starts from a fresh +context with no page open, so it calls `app.open()` first; on a device a test +that skips `app.open()` starts where the previous test left the app. + +## Registration + +`test` registers everything, and `describe`, `beforeEach`, `afterEach`, +`beforeAll`, and `afterAll` are also top-level imports of the same functions +(`import { describe, beforeEach } from 'e2e'`; `@e2e-dev/web` and +`@e2e-dev/mobile` export `test`, `describe`, and the hooks typed with their fixture; a +`test.extend()` chain registers hooks that see its fixtures through +`test.beforeEach`). Everything registers at import, so a `describe` body is +synchronous (`async` is a `COLLECTION_ERROR`). + +```ts +test('title', async ({ app, screen }) => {}); +test('title', { tags: ['smoke'], retries: 2, timeout: 60_000 }, async ({ app }) => {}); +describe('group', { tags: ['billing'] }, () => { /* tests and hooks */ }); +describe('checkout flow', { serial: true }, () => { /* ordered, shared app state */ }); +beforeEach(async ({ app }) => {}); // per attempt, with test fixtures +afterEach(async ({ screen }) => {}); // runs after failures too, with its own cleanup budget +beforeAll(async ({ platform }) => {}); // per suite realm, no app fixtures +afterAll(async () => {}); +test.skip('later', async () => {}); +test.only('focus', async () => {}); // local only: CI fails with ONLY_IN_CI +test('conditional', async () => { test.skip(await onlyOneOrg(), 'nothing to switch to'); }); // throws: the body stops here, reported skipped; call it before the first step +test('later', async () => { test.skip('waiting on the API'); }); // bare skip from the body +test.setup('sign in', { sessions: ['admin'] }, async ({ app, screen, session }) => {}); // see Sign-in sessions +const wsTest = test.extend<{ ws: Ws }>({ ws: async ({ browser }, use) => { await use(await seed()); await drop(); } }); +wsTest('uses the workspace', async ({ ws }) => {}); // code after use() is teardown, runs after failures too +``` + +A setup test cannot skip from its body (`INVALID_ARGUMENT`). + +| Option | Default | Notes | +| --- | --- | --- | +| `timeout` | `config.timeout`, 120 s | Covers `beforeEach` and the body. | +| `retries` | `config.retries` | 0 to 10; a serial group's value applies to its members. | +| `tags` | `[]` | Distinct, non-blank, no comma or edge whitespace (`'Login Form'` is fine); union across layers. `--tag smoke` selects, `--tag-mode all` requires every tag. | +| `skip` | unset | `true` or a reason string. | +| `platforms` | unset | Only targets with these platforms, e.g. `['ios']`. | +| `requires` | `[]` | Engine capabilities, e.g. `['browser']`; missing ones skip the test at selection rather than failing it with `UNSUPPORTED_CAPABILITY`. | +| `session` | unset | Restore state saved by a setup test. | +| `agentContext` | unset | Extra context for `agent.*` calls in this test or group. | +| `agent` | the run's agent | A configured name (`agents.`) or a list run once per agent; `--agent` narrows the list, a setup test takes one name. Innermost wins; `agent.act(..., { agent })` names another for one call. | +| `trace`, `video` | the target's | `'off'`, `'on'`, `'retain-on-failure'`, `'on-first-retry'`, `'on-all-retries'`: whether the test keeps a trace page, which attempts record video. Innermost wins over `--trace` / `--video`, the target, and the config; video where the engine cannot record is `UNSUPPORTED_ARTIFACT` for the run. | +| `serial` | `false` | Groups only. Members share app state, run in order on one worker, retry as a whole, and take the group's `trace` and `video`. | + +Serial members cannot set `retries`, `trace`, `video`, `session`, +`platforms`, `requires`, `skip`, or `only`, nor can a nested `describe` set +`trace` or `video`; nesting serial groups is a `COLLECTION_ERROR`. + +Hooks nest: outer `beforeEach` first, inner `afterEach` first. `beforeAll` +reruns per retry and per serial group (each a fresh module realm) on the +test timeout; `afterAll` runs on `config.cleanupTimeout`. A failing +`beforeAll` is a `HOOK_FAILED` run error skipping every test in its scope. + +## Fixtures + +Built-in fixtures are lazy; destructure them. Your own `test.extend` +fixtures set up for every test registered through that `test`, destructured +or not, so a state-changing fixture belongs on its own `test`. + +`app` (`App`) and `screen` (`Screen`): always. `agent` (`Agent`): needs a +configured model, else `MODEL_UNAVAILABLE` (topic `agent`). `platform` +(`string`): always, hooks included; `web`, `ios`, `android`, or an engine's +label. `browser` (`Browser`): browser targets, import `test` from `@e2e-dev/web`. +`device` (`Device`): device targets, import `test` from `@e2e-dev/mobile`. +`session` (`SetupSession`): only in `test.setup`. + +### app + +- `baseUrl` (`string | undefined`): the target's app URL with the run's + port; `undefined` when the target declares no `app.url`. +- `open(path?)`: opens the target's `app.url`, a relative path, or an absolute + http(s) URL. On a device it takes no path and relaunches the pinned app. +- `back()`: one history step back. +- `restart()`: recreates the context keeping persisted state (a restored + session included), then reopens the base URL. +- `clearState()`: clears cookies and storage, recreates the context, reopens + the base URL. +- `screenshot(label?)`: saves a redacted screenshot artifact and returns its + path. Denied after a secret fill (see Sign-in sessions). + +`clearState()` is `UNSUPPORTED_CAPABILITY` with `connect.reconnectEndpoint` +or an attempt-scoped browser provider; on a device with no pinned `app`, both +methods are. + +## Locators + +`screen.getBy*` builds a lazy query, resolved only by an action, read, or +assertion. Every query also exists on a locator, scoped to its subtree. + +| Query | Matches | +| --- | --- | +| `getByRole(role, name?, { exact?, checked?, disabled?, selected?, expanded?, pressed?, level?, visible? })` | Semantic role, optionally by accessible name (`getByRole('button', 'Save')`; the object form `{ name }` works too) and state (`level`: heading level 1 to 6). First choice. | +| `getByLabel(text, { exact?, visible? })` | Form controls by label. | +| `getByPlaceholder(text, { exact?, visible? })` | Inputs by placeholder. | +| `getByText(text, { exact?, visible? })` | Visible text. | +| `getByDisplayValue(value, { exact?, visible? })` | Inputs by current value; on the web it cannot scope child queries or be a `filter({ has })` target. | +| `getByTestId(id, { visible? })` | `data-testid` on the web (or `web({ testIdAttribute })`), accessibility identifier or resource id on a device (React Native `testID`, SwiftUI `.accessibilityIdentifier`, Compose `testTag` with `testTagsAsResourceId` on Android, Flutter `Semantics(identifier:)`); a string matches the whole id, a RegExp tests it. Last resort. | + +Roles: `button`, `link`, `textbox`, `searchbox`, `combobox`, `listbox`, +`option`, `checkbox`, `radio`, `radiogroup`, `switch`, `slider`, `spinbutton`, +`progressbar`, `meter`, `image`, `heading`, `tab`, `tablist`, `tabpanel`, +`menu`, `menubar`, `menuitem`, `menuitemcheckbox`, `menuitemradio`, `toolbar`, +`tooltip`, `tree`, `treeitem`, `list`, `listitem`, `table`, `grid`, `row`, +`rowgroup`, `rowheader`, `cell`, `gridcell`, `columnheader`, `separator`, +`group`, `article`, `figure`, `form`, `status`, `alert`, `dialog`, +`alertdialog`, `main`, `navigation`, `banner`, `contentinfo`, `complementary`, +`region`. The union is closed (anything else is a type error); `img` aliases +`image`; a role the platform lacks (`tooltip` on a phone) matches nothing. +On the web a `contenteditable` host is a `textbox` for the agent and takes +`fill`; from a test use `getByLabel`, `getByTestId`, or `role="textbox"` on +the host. + +Text matching is exact after whitespace normalization (whole string, +case-sensitive) for a `getByRole` name, `getByLabel`, `getByPlaceholder`, and +`getByText`: `name: 'Save'` misses `Save changes` (unlike Playwright). A +missed query makes a negated assertion pass, so check it positively. +`getByText` and `getByLabel` return the innermost match, so a container +echoing its child does not count twice. `exact: false` is a case-insensitive +substring; a `RegExp` matches as written. + +- Exactly one match per action, read, or assertion. Two fail at once with + `LOCATOR_AMBIGUOUS`; at zero, actions and assertions poll until the + timeout then `LOCATOR_NOT_FOUND`, a read fails at once. Exceptions: + `toHaveCount`, `toBeVisible`, `toBeHidden`, `toBeAttached`, list-form + `toHaveText` and `toContainText`, `isVisible()` (false at zero), + `isHidden()` (true), `count()`, `all()`, `allTextContents()`. +- Narrow with `filter({ hasText })` (a case-insensitive substring, unlike a + query), `filter({ has: locator })`, `first()`, `last()`, `nth(i)`, or + scoping under another locator. Any other `filter` key (`hasNot`, + `hasNotText`) or an empty `filter({})` is `INVALID_LOCATOR`. +- `visible: true` drops nodes the page hides (a closed drawer) before the + exactly-one rule. Visibility is what renders, as `toBeVisible()` reads it: + an `aria-hidden` spinner that paints is visible, so `toBeHidden()` waits + for it to go. +- On the web, queries reach open shadow roots and closed roots attached with + `attachShadow`, not declarative closed roots. `browser.locator(css)`, + `frameLocator`, and `filter({ hasText })` stop at a closed root; query the + text inside or filter with `has`. +- `screen.scrollUntilVisible(locator, { direction?, momentum?, timeout? })` + scrolls the viewport (`down`, `slow` by default) until the locator resolves + visibly, else `LOCATOR_NOT_FOUND`; on a locator it scrolls that node. + +### Actions + +Each action resolves one node, waits up to `config.actionTimeout` (30 s, or +`{ timeout }`) for it to be actionable, and does one thing: `tap()` (alias +`click()`), `doubleTap()` (alias `dblclick()`), `secondaryTap()` (each takes +`{ modifiers: ['Shift'] }`), `longPress({ duration? })` +(100 to 10000 ms; default 500 on the web, 1000 on a device), +`fill(value | Secret)`, `pressSequentially(text, { delay? })`, `clear()`, +`press(key)`, `check()`, `uncheck()`, +`selectOption(label | { label } | { value } | { index })` (one option; an +array is `INVALID_ARGUMENT`), `focus()`, `hover()`, `setInputFiles(paths)` +(relative to the project root), `dragTo(locator)`, `scrollIntoView()`, +`swipe({ direction, momentum? })`. Web only, `UNSUPPORTED_CAPABILITY` on a +device: `secondaryTap`, `selectOption`, `setInputFiles`, `scrollIntoView`, +and `modifiers`. + +`fill` sets the value with no key events; when the app reacts to keystrokes +(autocomplete, a masked input) use `pressSequentially`, which focuses the +field and types one character per `delay`, plain string only (a `Secret` is +`INVALID_ARGUMENT` and goes through `fill`). + +Coordinates are CSS pixels, for what the tree does not list: `tap({ position: +{ x, y } })` offsets from the node's top-left corner (with `modifiers` it is +`INVALID_ARGUMENT`), `screen.tapAt({ x, y })` taps a viewport point, +`screen.swipe({ from, to, duration? })` swipes along a path, +`screen.swipe({ direction, momentum? })` swipes the viewport. Prefer a +locator; a point moves with the layout. + +### Reads + +Reads resolve once, no retry: `textContent()`, `inputValue()`, +`getAttribute(name)`, `isVisible()`, `isHidden()`, `isEnabled()`, +`isDisabled()`, `isChecked()`, `boundingBox()`, `count()`. `all()` gives one +`nth(i)` locator per current match and `allTextContents()` every match's +text, both `[]` at zero. Text is the rendered text, whitespace collapsed: on +the web what `innerText` reads (`text-transform` applies, `display: none` +drops out, `
` is a space); `toHaveText` reads the same. `isChecked()` is +`false`, not an error, on a node with no checked state, so query checkable +controls by role. `waitFor({ state?: 'attached' | 'detached' | 'visible' | 'hidden', timeout? })` waits +within `actionTimeout`, else `LOCATOR_NOT_FOUND`. For a value that has to +settle use `expect`, not a read. Reading a password field's value or +attributes is `POLICY_DENIED`, as is `toHaveAttribute` on one, negated too. + +## expect + +`expect(locator)` polls up to `config.assertionTimeout` (5 s) or +`{ timeout }`; `.not` inverts and passes once the negation has held 1 s +continuously, or for the whole `timeout` when that is shorter, so it never +returns sooner. `expect(value, +message?)` is synchronous. +`expect.poll(read, { timeout?, interval?, message? })` re-reads until a value +matcher passes (`assertionTimeout` and 100 ms by default, stopping with the +attempt); a throwing read keeps polling, and it is not a report step. A +poll that the body or a hook returns without awaiting is cancelled and fails +that phase with `STEP_NOT_AWAITED`. +`expect.soft(x)` keeps a failure instead of throwing; the attempt fails +after the body with every soft failure listed. If the body then calls +`test.skip`, the attempt still fails, with the skip reason in its `skip`. +A retry that skips after an earlier failed attempt stays skipped; the CLI shows +it under `Skipped After Failure`, and `failOnSkippedFailure: true` in the +config fails the run with exit code 1. The default is `false`. +`expect.any(Class)`, `expect.anything()`, `expect.objectContaining(obj)`, +`expect.arrayContaining(arr)`, `expect.stringContaining(s)`, and +`expect.stringMatching(s | RegExp)` stand in for values inside `toEqual`, +`toMatchObject`, `toContain`, and `toHaveProperty`. + +```ts +await expect(screen.getByRole('dialog')).not.toBeVisible({ timeout: 10_000 }); +await expect(browser).toHaveURL('/dashboard'); // relative to the base URL, or a RegExp +expect(order).toMatchObject({ id: expect.any(Number), lines: [{ sku: 'a' }] }); +expect.soft(await screen.getByTestId('tax').textContent()).toBe('$8.00'); // kept, body runs on +const users = expect(await response.json()).toMatchSchema(z.array(User)); // any Standard Schema; typed output +``` + +`toMatchSchema(schema)` takes a synchronous Standard Schema (Zod, Valibot, +ArkType), fails listing every issue by path, and returns the parsed value +typed; prefer it when the app already has a schema for the response. + +| Locator matchers | Browser matchers | Value matchers | +| --- | --- | --- | +| `toBeVisible`, `toBeHidden`, `toBeAttached`, `toBeEnabled`, `toBeDisabled`, `toBeChecked`, `toBeSelected`, `toBeExpanded`, `toBeFocused`, `toHaveText`, `toContainText`, `toHaveValue`, `toHaveAttribute`, `toHaveCount`, `toHaveAccessibleName`, `toHaveScreenshot` (also on `screen`) | `toHaveURL`, `toHaveTitle`, `toHaveClass(locator, expected)` | `toBe`, `toEqual`, `toMatchObject`, `toBeTruthy`, `toBeFalsy`, `toBeNull`, `toBeUndefined`, `toBeDefined`, `toHaveLength`, `toHaveProperty`, `toContain`, `toMatch`, `toBeGreaterThan`, `toBeGreaterThanOrEqual`, `toBeLessThan`, `toBeLessThanOrEqual`, `toBeCloseTo`, `toMatchSchema` | + +`toHaveText` compares the whole normalized text, `toContainText` a substring +or RegExp, `toHaveValue` a form control's value as is, whitespace included +(it fails on a node with none); on a password field all three are +`POLICY_DENIED`, never a comparison against `''`. List forms: +`toHaveText(['Alpha', /^Beta/])` needs exactly two matches with those texts +in order; `toContainText(['Alpha', 'Beta'])` needs each entry in a distinct +match, in order, extra matches allowed. `toHaveAttribute(name)` checks +presence, `toHaveAttribute(name, value)` the value; `toBeAttached` waits for +a match, hidden or not; `toHaveClass` compares the whole normalized class +list or tests a RegExp. A failed matcher is `ASSERTION_FAILED`, exit code 1. + +Playwright's `{ checked: false }`, `{ enabled: false }`, `{ visible: false }`, +and `{ attached: false }` flip their matchers, and `{ ignoreCase: true }` +works on `toHaveText`, `toContainText`, `toHaveAccessibleName`, +`toHaveAttribute(name, value)`, and `expect(browser).toHaveURL`. Any other option, `indeterminate` or +`useInnerText` included, is `INVALID_ARGUMENT`, in JavaScript too. + +Screenshots: `await expect(screen).toHaveScreenshot('home.png')` compares the +screen, `expect(locator).toHaveScreenshot()` one element's box, against +`-snapshots/--.png`, on web, iOS, and Android +alike. Commit the files. The first run writes the file and fails: look at it, +commit it, rerun. A CI run writes nothing into the project and attaches a +missing one at `snapshots/` in the test's results. `--update-snapshots` +(`-u`) writes missing and different files and passes, after an intended UI +change. Options: `threshold`, `maxDiffPixels`, `maxDiffPixelRatio`, `mask` +(locators painted over, for clocks and other changing content), `maskColor`, +`timeout`. A failure attaches `-expected`, `-actual`, and `-diff` images to the +step. Files are per target and OS (`-web-linux.png`): screenshots written on a +Mac never serve Linux CI, so write CI's from a CI job with `-u`. + +## Sign-in sessions + +Sign in once in a setup test, save the state under a name, and let other +tests declare it. Selecting a dependent test alone still runs its setup. + +```ts +// tests/auth.setup.e2e.ts +import { test } from '@e2e-dev/web'; +import { expect, credentials } from 'e2e'; + +test.setup('authenticate as admin', { sessions: ['admin'] }, async ({ app, screen, session, browser }) => { + const admin = credentials.user('admin'); + await app.open('/login'); + await screen.getByLabel('Email').fill(admin.username); + await screen.getByLabel('Password').fill(admin.password); + await screen.getByRole('button', 'Sign in').tap(); + await expect(browser).toHaveURL('/dashboard'); // prove the sign-in worked before saving + await session.save('admin'); +}); +``` + +```ts +// tests/dashboard.e2e.ts +import { test, expect } from 'e2e'; + +test('the dashboard opens directly', { session: 'admin' }, async ({ app, screen }) => { + await app.open('/dashboard'); + await expect(screen.getByRole('heading', 'Dashboard')).toBeVisible(); +}); +``` + +- Setup tests are top-level; exactly one setup saves a given name and the + body saves every declared name once (`SESSION_CONTRACT` otherwise). Names + match `[A-Za-z0-9_.-]{1,128}`. +- A session holds cookies, local storage, and IndexedDB for one run, + encrypted and deleted at cleanup; server state is not part of it. +- Once a secret is filled, model pixels and assertion screenshots are + withheld for the rest of that session (later serial members included) and + `app.screenshot()` is `POLICY_DENIED`. A restored session keeps its setup's + taint; a setup that signs in without a fill (`browser.setCookies`, say) leaves + screenshots available. +- Credentials live in the config, values in the environment: + +```ts +credentials: { + admin: { username: 'admin@example.test', password: process.env.ADMIN_PASSWORD ?? '' }, +}, +``` + +- A static password or secret needs 6 or more code points, else + `INVALID_CONFIG` at config load, so an unset variable fails every command. + A function `password` is read at fill time and redacted only from that fill + on; `username` is always a string. +- `E2E_USER__USERNAME` and `E2E_USER__PASSWORD` override either + field per run, even over a function; `` is the credential name + uppercased, every character outside `[A-Z0-9]` as `_`. Two credentials (or + two secrets) mapping to one variable, `svc-a` and `svc_a`, are + `INVALID_CONFIG`. +- `credentials.user('admin').password` is a `Secret` with no plaintext + accessor, named `admin.password` to the agent and in reports; `secrets.get()` + never returns it (separate namespaces). Only `fill()` and `agent.act` params accept it, stringifying it + is `INVALID_CONFIG`, and `credentials.user()` outside a run throws + `AUTH_CREDENTIAL_UNAVAILABLE`. +- Any other sensitive value (an API key) is a `secrets` entry, + `secrets: { 'stripe-key': process.env.STRIPE_KEY ?? '' }`, overridable with + `E2E_SECRET_STRIPE_KEY`; `secrets.get('stripe-key')` is the same kind of + handle, fills any editable input, and is redacted by name everywhere the + runner writes. +- A registered value passed as a plain string (`process.env.STRIPE_KEY` in a + title, a URL, a locator, an `agent.act` instruction or param) is redacted + too: reports, step labels, the executor, and the model see + `` (and so does the test id, for a value in a title), + so the agent cannot type it. Pass the handle. + +## The browser fixture (browser only) + +Prefer `app` and `screen`; `browser` is for what only a browser has, and a +portable suite declares `requires: ['browser']`. Page methods act on the one +active tab; cookies, routes, and `onDialog` cover the whole browser. A tab +the app opens itself (`target="_blank"`, `window.open`) is not followed: +`browser.goto` its URL instead. + +- `goto(url, { waitUntil?, timeout? })`, `reload({ timeout? })`, + `back({ timeout? })`, `forward({ timeout? })`: navigation on the test + timeout by default; `goto` takes a base-relative path. +- `url()`, `title()`, `waitForURL(url | RegExp, { timeout? })`: reads and a + URL wait; `waitForURL` matches like `toHaveURL`. +- `locator(css)`: raw CSS or XPath; not portable, a last resort. +- `frameLocator(css)`: a `Screen` scoped to one iframe + (`browser.frameLocator('#payment').getByLabel('Card number')`); keeps + `locator(css)` and `frameLocator(css)` for nesting. +- `evaluate(fn | source, arg?)`: runs a function or source string in the + page, JSON in and out, no closures; a throw in the page is + `EVALUATE_FAILED`. +- `addInitScript(source | { path } | fn)`, `addInitScript(fn, arg)`: runs + before the page's own scripts; call it before `app.open`. `arg` is JSON. +- `route(pattern, handler)`, `unroute(pattern)`: intercept requests, newest + route first; `route.request` has `url`, `method`, `headers`, `postData`. + The handler calls exactly one of + `fulfill({ status?, headers?, contentType?, json | body | path })`, + `continue({ url?, method?, headers?, postData? })` (straight to the + network), `fallback()` (the route registered before it), or `abort()`; + none or two fails the next step with `ACTION_FAILED`, an unsupported + option with `INVALID_ARGUMENT`. +- `waitForResponse(pattern, { timeout? })`: resolves once the headers + arrive, with `{ url, status, headers, json(), text() }`; `text()` and + `json()` wait for the body (up to the action timeout) and reject with + `ACTION_FAILED` when it could not be read. +- `cookies()`, `setCookies([...])`: a target is an http(s) URL or a domain. +- `setViewport({ width, height })`: resize. +- `onDialog('accept' | 'dismiss' | handler)`: awaited; resolves to an async + unsubscribe. Register it before the tap that opens the dialog. A handler + gets `{ message, accept(text?), dismiss() }`, `accept` taking the prompt + text; no handler, or one that neither accepts nor dismisses, fails the + next step with `INVALID_STATE`. A throwing handler's unanswered dialog is + dismissed too, and its error fails the next step. +- `waitForDownload(() => trigger, { timeout? })`: returns + `{ path, suggestedFilename }`. +- `keyboard.press(key)`, `keyboard.type(text)`, `mouse.*`: unfocused input; + prefer `locator.press` and `locator.fill`. + +A `route`, `unroute`, or `waitForResponse` pattern is a glob string or +`RegExp` matched against the full URL (`*` stays within one path segment, +`**` crosses `/`, `?` is one character); a predicate function is +`INVALID_ARGUMENT`. + +## Habits + +- Selectors come from the source (labels, roles, text); add an `aria-label` + or heading where the app has no accessible name rather than fall back to + `browser.locator('.btn-primary')`. +- Test data gets a run-unique name (`Invoice ${Date.now()}`) and `afterEach` + cleanup, so replays and retries never trip over leftovers. +- APIs live in the same suite: `fetch(new URL('/api/users', app.baseUrl))` + plus value matchers, in a `test.extend` fixture that reads `browser.cookies()` + when the API needs the session. +- No sleeps or polling loops; a matcher with a longer `timeout` instead. +- `await` every step call and `expect.poll`, else `STEP_NOT_AWAITED` at the line of the call. +- Assert the fact a model produced with `toContain`, not its exact sentence. diff --git a/log/changelog-v2.5.0.md b/log/changelog-v2.5.0.md new file mode 100644 index 0000000..56a0de7 --- /dev/null +++ b/log/changelog-v2.5.0.md @@ -0,0 +1,54 @@ +# v2.5.0 — optional e2e runner adapter, carried inside the kit + +v2.4.0 made QA save a script per scenario so later rounds re-run instead of +re-driving by hand. This adds a runner those scripts can be written for, +tried first on one feature's four browser scenarios against a dev server. + +## Changes + +- **`adapters/e2e/`.** With config `qa_runner: e2e`, QA writes each browser + or mobile scenario as an [e2e](https://www.npmjs.com/package/e2e) test in + two layers tagged with the scenario ID: `det` (exact steps, locator, URL + and request assertions, no model) decides PASS or FAIL; `agent` (optional) + uses `agent.act` for churning steps and replays without model calls on + later runs. Measured on the trial: `det` 24.6–25.9 s per run, 0 model + calls, 2 of 2 runs green; `agent` 47.8 s and 10 model calls on an empty + cache, then 32.6 s and 2 calls (both `agent.assert`, which never caches) + with 3 of 3 act steps replayed. A hand-driven script covering a wider + scope took 292 s. +- **Portable.** The kit vendors e2e's own agent guide (`skills/e2e`, + Apache-2.0, `LICENSE` and `NOTICE` kept) pinned in `adapters/e2e/VERSION` + (e2e 0.20.0, @e2e-dev/web 0.14.0, Node `^22.22.3 || >=24.8.0`), so an + agent can write and debug tests offline. `adapters/e2e/update.sh + [version]` refreshes it from npm. The project installs the runner itself. +- **`qa_session:`.** e2e does not follow windows an app opens, so a sign-in + through a popup, SSO or a device prompt cannot be driven. `qa_session` + names a command that signs in and writes the storage the setup template + injects before saving a named session. +- **Popup bridge.** e2e does not follow windows an app opens. The template + `popup-bridge.ts` turns `window.open` into a full-viewport iframe in the + same tab, with `window.opener`, `close()` and `focus()` working inside it, + so same-origin popups that talk through opener and postMessage run + unchanged. Tried on two document-viewer popups: deterministic run 25.0 s + and 24.1 s, 4 of 4 both times; an agent step inside the popup took 2 model + calls (12.3k tokens) first and 0 on replay. Screenshots matched real + popups except window size. Not covered: cross-origin popups, + `target="_blank"` links, `noopener`. +- **Opening can write.** In that trial, opening a one-page assignment in its + viewer marked it submitted. QA's preflight now treats no step as read-only + until it knows what the step writes, and drives only accounts whose state + may change. +- **Templates.** `e2e.config.ts`, `session.setup.e2e.ts`, + `scenario.e2e.ts` (both layers) and `popup-bridge.ts`, checked with `e2e list` and `tsc`. +- **Limits written down from the trial.** An agent step absorbs a renamed + control and still passes, so the `det` locator stays the gate. A wrong + expectation after a correct replay evicts the correct recording. A replay + that cannot find its target waits before handing over and counts as + missed. Sessions keep cookies and localStorage, not sessionStorage. The + replay cache stores typed inputs in plain text. Agent steps use a + subscription login made once by the human; telemetry is turned off with + `E2E_TELEMETRY_DISABLED=1`. +- `roles/qa.md` reads the adapter when `qa_runner` is set and runs + `qa_session` in its preflight; config template, SKILL.md and both READMEs + name the two keys. Selftest checks the adapter's files, pinned version + and updater (120 checks). diff --git a/roles/qa.md b/roles/qa.md index 0238d0c..98e856a 100644 --- a/roles/qa.md +++ b/roles/qa.md @@ -10,7 +10,10 @@ commit(s) to verify. one real call through each route the scenarios use reaches the intended environment (proxy, host, account); the driver starts in this sandbox; anything long-running is started detached with a limit longer than the - run. Fix what you may, or mark the scenarios it blocks BLOCKED with the + run; config `qa_session`, if set, signs in and its setup check passes. + Treat no step as read-only until you know: opening, closing or paging + through a record can write on the server. On shared data, find what each + step writes first, and drive only accounts whose state may change. Fix what you may, or mark the scenarios it blocks BLOCKED with the cause, and put what you learned in `refs/learnings.md`. 1. Confirm the checked-out commit is the one you were given. Run `build`, `test`, `lint`, then `e2e` from config.md. A failure the base commit shows @@ -22,7 +25,9 @@ commit(s) to verify. library call. Read state back after every write. Save the raw output (command, response, screenshot path, query result) to `logs/.txt`, and the script or command sequence that drove it to `logs/scripts/`, so - a later round re-runs it with one command. + a later round re-runs it with one command. With config `qa_runner: `, + the script is that runner's test: read `adapters//README.md` in the + kit folder first; its deterministic layer decides PASS or FAIL. 4. A scenario that cannot run here is BLOCKED, with what is missing. Never PASS by assumption; never a unit test in place of the real interface. 5. Bug fix: run the regression test at the commit before the fix, in a diff --git a/templates/config.md b/templates/config.md index 6e60e53..1a8c373 100644 --- a/templates/config.md +++ b/templates/config.md @@ -10,6 +10,10 @@ run: e2e: # how QA drives the interface: a browser driver, an HTTP client, the shell; empty = whatever the agent has qa: +# optional scenario runner adapter in the kit's adapters/ folder (e2e); empty = scripts in logs/scripts/ +qa_runner: +# command that signs in for QA where the runner cannot (popup, SSO) and writes the session file it injects; empty = none +qa_session: # gates a human approves: intent, plan. Plan also needs the human when PLAN.md raises a risk flag. human_gates: intent # durable knowledge: scouts read it first, Evidence adds verified facts to it diff --git a/tests/selftest.sh b/tests/selftest.sh index 7cc2e33..1342111 100755 --- a/tests/selftest.sh +++ b/tests/selftest.sh @@ -1459,4 +1459,10 @@ assert_file .sdlc/archive/count-todos/evidence.md assert_file .sdlc/archive/count-todos/refs/data.md expect 30 next "$SLUG" ok 'real four-phase lifecycle: old CLI data, scouts, plan, change, commit, review, four CLI scenarios, score, local archive' +# The e2e adapter is self-contained: vendored guide with its license, a pinned VERSION, a runnable updater. +A=$KIT/adapters/e2e +for f in README.md VERSION update.sh vendor/e2e-skill/SKILL.md vendor/e2e-skill/LICENSE vendor/e2e-skill/NOTICE templates/e2e.config.ts templates/session.setup.e2e.ts templates/scenario.e2e.ts templates/popup-bridge.ts; do assert_file "$A/$f"; done +awk -F': *' '$1=="e2e"&&$2~/^[0-9]+\.[0-9]+\.[0-9]+$/{ok=1}END{exit !ok}' "$A/VERSION" || fail 'adapters/e2e/VERSION has no pinned e2e version' +bash "$A/update.sh" --help > "$TMP/update-help" || fail 'adapters/e2e/update.sh --help failed' +ok 'e2e adapter: vendored guide, license, pinned version, templates, updater help' printf 'PASS: %s checks\n' "$COUNT"