five46

Say what should work, in plain English. An AI agent checks it against your real app or API, using your own LLM key, and hands you a real Playwright test.

You just built something and want to know whether it actually works, before writing a test. Give five46 a goal like "log in and confirm the dashboard loads" or "create a user via POST, then confirm it via GET". An LLM (OpenAI, Anthropic, Gemini, Groq or AWS Bedrock, with your key) drives your real app or API one action at a time. It tells you honestly whether the goal was met, with a root-cause hypothesis when it wasn't. A successful run is saved as a standalone Playwright (or node:test) spec, so today's check becomes tomorrow's regression test, and running it needs no five46 or LLM.

Quick start

npm install -g five46
npx playwright install chromium   # one-time browser download
five46 config                     # pick a provider and paste a key (Gemini's is free)
five46 doctor                     # confirms the key works
five46 test http://localhost:3000 --goal "log in as demo@example.com and confirm the dashboard says Welcome back"

You get a pass or fail (and a matching exit code), a report with screenshots and, for failures, a root-cause hypothesis, plus a five46-agent-*.spec.ts you can commit and run with npx playwright test. Testing an API instead? Use five46 api <base-url> --goal "...". It's read-only unless you pass --allow-writes.

Status: early and actively developed. Verified end-to-end against dozens of real-world sites and APIs.

Check every pull request

The five46 GitHub Action reads the Acceptance criteria section of a pull request's description and checks each item against the PR's preview deployment. It keeps one PR comment up to date with the verdict per criterion, why each failure failed, and a ready-to-commit spec for every criterion that passed.

on: deployment_status   # Vercel, Netlify and other preview deploys
jobs:
  five46:
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    permissions: { contents: read, pull-requests: write }
    steps:
      - uses: sekharsdet/five46-action@v1
        with:
          url: ${{ github.event.deployment_status.target_url }}
          api-key: ${{ secrets.FIVE46_API_KEY }}

How it compares

	five46	Playwright Test Agents	playwright codegen	Cloud AI testing platforms
You provide	A goal or acceptance criteria in plain English	A seed test and a request, inside an AI coding tool	Your own clicks	Test descriptions in their app
Runs	CLI, CI, GitHub Action or MCP tool	Inside an AI coding tool, such as VS Code	Locally, while you click	Their cloud
Built to give a pass/fail verdict on the app	Yes, with a root-cause hypothesis	Not its focus: it generates and repairs tests	No	Yes
API testing	Yes (node:test output)	Browser-focused	No	Varies
Proves assertions catch failures	Yes (--verify mutation check)	No built-in check	No	Varies
LLM and cost	Your key, any of 5 providers	Your coding agent's model	None	Their subscription
Your app data goes to	Only your LLM provider	Your coding agent's provider	Nowhere	Their platform
Output	Plain Playwright or node:test files	Plain Playwright files	Plain Playwright files	Usually their runner

Already working in an AI coding tool and growing a Playwright suite there? Playwright's own agents are a great fit, especially the healer for keeping existing tests green. five46 is for getting a verdict: is this feature working right now, in CI or on a PR, for a browser flow or an API, with any LLM you already pay for. More detail is in Why five46.

Highlights

• Safe by default. API runs are read-only unless you unlock writes; destructive-looking clicks ("Delete account") are blocked unless allowed.
• Login credentials never reach the LLM. It sees placeholders, and the real values are filled in locally. Log in once, then reuse the session.
• Acceptance criteria in, verdicts out. --story splits a user story or ticket into independent scenarios and reports pass/fail per criterion.
• Specs you can trust. --verify-clean-session re-runs the spec cold; --verify mutates its assertions and requires them to fail.
• Works with any current key. If your provider retires a model, five46 finds a current one your key can use. You can also pin one with FIVE46_LLM_MODEL.
• Fits your tools. CI-friendly exit codes, --report JSON for dashboards, flaky-run detection (--repeat), video recording, and an MCP server for IDE assistants.

Docs

• Configuration: providers, getting a key, five46 doctor, choosing models
• CLI reference: every command and flag, login sessions, API testing, story mode, verification, reports
• MCP server: use five46 from an IDE-embedded AI assistant
• Security model: what leaves your machine, what five46 guards, and what's your call
• Why five46: design choices, the full feature list, comparisons
• CHANGELOG

Known limitations

• No closed shadow DOM traversal. Elements inside a mode: "closed" shadow root aren't visible to the agent's page snapshot (a closed shadow root is deliberately inaccessible to external tooling in general, not just five46). Open shadow roots, the overwhelming majority of real-world web components, work fine, as do <iframe> and <frame> content.
• Chromium only for browser mode, with no Firefox or WebKit yet.
• A run itself is not deterministic. The same goal against the same page can take a different path next time. The generated spec is the frozen, repeatable artifact.

Contributing

Issues and pull requests are welcome. See CONTRIBUTING.md.

git clone https://github.com/sekharsdet/five46.git
cd five46
npm install
npm run build
node dist/cli.js test <url> --goal "..."
npm test

If five46 is useful to you, a ⭐ on GitHub helps other people find it.

License

MIT