Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 17 additions & 3 deletions .agents/skills/e2e/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,17 @@
---
name: e2e
description: Writes, runs, and debugs end-to-end tests for web and mobile apps with e2e, the agentic test runner, which mixes agent.act goals with exact locator assertions on a Playwright browser or an iOS or Android device. Also drives a project's running app live over the e2e MCP server to look at a screen, check a UI change, take a screenshot or record a video, and try a locator; and runs bug bashes, parallel e2e explore runs with each bug proven by a repro test. Use when a project depends on e2e or has an e2e.config.ts, when asked for end-to-end, browser, mobile, or agentic UI tests, to check a change in the running app, to QA or bug bash an app or branch, or when an e2e run fails.
metadata:
openclaw:
homepage: https://e2e.com
requires:
bins:
- node
- npx
envVars:
- name: AI_GATEWAY_API_KEY
required: false
description: Model key for agent steps when the project's e2e config uses the Vercel AI Gateway; other providers read their own key.
---

# e2e: agentic end-to-end tests
Expand Down Expand Up @@ -108,8 +119,10 @@ test('a member upgrades to Pro', async ({ app, agent, screen, browser }) => {
Exact values go through `screen`: a sign-in form in a setup test, a field
that must receive one specific string, a count that must be one number.
4. Run only the test you are working on:
`npx e2e run tests/<feature>.e2e.ts:<line>`, the line its `test(` opens
on, with the flags in "Rerunning a test while you fix it" below. Agent
`npx e2e run tests/<feature>.e2e.ts:<line>` (over MCP,
`run_tests {files: ["tests/<feature>.e2e.ts:<line>"]}`, the flags as
camelCase fields such as `retries: 0`), the line its `test(` opens on,
with the flags in "Rerunning a test while you fix it" below. Agent
steps need a model in the config and that provider's authentication (a
saved subscription login, an API key); a local endpoint may need none.
Tests without agent steps need no model.
Expand Down Expand Up @@ -195,7 +208,8 @@ lines you changed:
- In a project without e2e, set it up when the user requests it. Otherwise
ask before adding a test framework, and name any existing framework.
- Live MCP tools need the project's own `e2e mcp` server registered in the
client (topic `mcp`); writing and running tests use the CLI directly.
client (topic `mcp`). Tests run with the CLI, or with the server's
`run_tests`, the same run without a shell.
Keep a session open across runs while you write or fix a web test; topic
`mcp` says when a run can go beside it, and what to do on
`APP_ALREADY_RUNNING`.
Expand Down
17 changes: 12 additions & 5 deletions .agents/skills/e2e/references/agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,14 +49,14 @@ under `executor`. `openai.decisionModel('gpt-6-luna')` from `@ai-sdk/openai`
4.0.90 or later works too and takes the `vision: true` screenshot.
Tests stay plain language with no params. The text model writes field values
and upload paths. See the shipped `docs/decision-models.mdx` or
[the online guide](https://e2e.tester.army/docs/decision-models) for setup,
[the online guide](https://e2e.com/docs/decision-models) for setup,
gates, and limits.

A coding agent the user already signs in to runs agent steps over ACP:
A coding agent the user already signs in to runs agent steps over the Agent Client Protocol:
install `@e2e-dev/acp` and its adapter, and put
`claudeCode({ model: 'sonnet' })` or `codex()` under
`executor`; no model provider is needed. See the shipped `docs/acp.mdx` or
[the online guide](https://e2e.tester.army/docs/acp).
`executor`; no model provider is needed. See the shipped `docs/subscriptions.mdx` or
[the online guide](https://e2e.com/docs/subscriptions#use-your-coding-agent).

- Pass a model instance, not a string (`INVALID_CONFIG`).
- An agents entry is one plain object of `model`, `judge`, `system`,
Expand Down Expand Up @@ -154,6 +154,9 @@ The tools, one per engine action. Name the target as the screen names it
native file chooser; outside it or hidden (`.env`) is `POLICY_DENIED`.
- `navigate`: a URL or app-relative path.
- `back`: browser history or in-app back.
- `resize_viewport`: set the viewport size (200 to 4096 CSS px) to check a
layout at phone, tablet, or desktop width. Engines with `session.resize`,
such as web.
- `type_secret`: a declared secret by name; plaintext never reaches the model.
Offered only when the step declares secrets and the engine can fill them.
- `screenshot`: attach viewport pixels; every later result then carries one.
Expand Down Expand Up @@ -241,6 +244,10 @@ For a debug view that pairs each action with the screen it started on, give
each capture to `onAction` by `index`, which restarts at 1 in each `act` call.
It runs before replayed actions too, does not wait for the screen to settle,
and a failed action gets a capture but no `onAction`.
Narrow `onAction`'s event on `action` for coordinates, replay included:
`point` for `tapAt`/`hoverAt`, `from`/`to` for `dragAt`. They are in the
same space as `screen.tapAt` and `boundingBox()`, rounded and clamped to the
viewport. Mobile engines have no `hoverAt` or `dragAt`.

`vision` on a judgment picks the evidence: `false` (default) the tree;
`true` the tree plus a masked screenshot; `'only'` the screenshot alone, for
Expand Down Expand Up @@ -428,7 +435,7 @@ make no model calls.
`executor`; the runner still owns observations, actions, budgets, and the
report, and `system` or `tools` beside `executor` is `INVALID_CONFIG`.

Reference: https://e2e.tester.army/docs/agents
Reference: https://e2e.com/docs/agents

## In CI

Expand Down
2 changes: 1 addition & 1 deletion .agents/skills/e2e/references/debugging.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ default.
timeout, changing values at a race;
- what the app logged meanwhile (`✗ network error: POST /api/save 500`,
`✗ uncaught error: ...`, `⚠ console warning: ...`, `ℹ console: ...`);
`❯ app 2 errors` in the terminal says there are some, and the page's
`❯ app 2 errors` in the terminal quotes the first 3 errors, and the page's
`## App log` lists up to 50 with their step, errors and warnings
first, counting the rest;
- where the page went (`↪ navigated to /login`, a new tab the app
Expand Down
4 changes: 2 additions & 2 deletions .agents/skills/e2e/references/explore.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,8 +72,8 @@ configured secrets are redacted (topic `writing-tests`).
| `--target <id>` | first configured target | The one target to explore. |
| `--agent <name>` | `default` | Build the explorer from another configured agent (`agents.<name>`). |
| `--session <name>` | none | Run the setup that saves this session, then explore with it restored. |
| `--max-steps <n>` | 8 (1 to 12) | Exploration steps at most. |
| `--timeout <ms>` | 600000 (180000 to 900000) | Wall clock; the last minute is for the assessment. |
| `--max-steps <n>` | 8 (1 to 100) | Exploration steps at most. |
| `--timeout <ms>` | 600000 (180000 to 7200000) | Wall clock; the last minute is for the assessment. |
| `--headed`, `--reporter`, `--output`, `--debug`, `--ai-trace`, `--trace [mode]`, `--video [mode]` | as `run` | Same meaning as for `e2e run`. The exploration is one attempt, so a retry mode (`on-first-retry`, `on-all-retries`) records nothing (the run's notice says to pass `--trace on`); put the goal before a bare `--trace` or `--video`. |

Per-step action and model-call budgets default to 40 each;
Expand Down
Loading
Loading