Skip to content

cli-reference.md

Latest commit

 

History

History
335 lines (274 loc) · 14.3 KB

cli-reference.md

File metadata and controls

335 lines (274 loc) · 14.3 KB
Raw

CLI reference

← Back to the README

Running a test

five46 test http://localhost:3000 --goal "log in and confirm the dashboard loads"

Exactly one of --goal or --story is required. Useful flags: --max-steps (default 15), --headed (watch it drive a real visible browser instead of headless), --out (spec path), --allow-deletes (allow clicking destructive-looking elements, e.g. "Delete Account"), --no-root-cause (skip the extra LLM call that analyzes a failed assertion), --repeat N (run the goal N times and report whether it's flaky — see below), --record-video (save a .webm of the whole session), --project name (pull defaults from five46.config.json — see below), --no-structured-plan (opt out of the default upfront-plan-then-fast-path behavior and use the fully-adaptive, live-decision-every-step loop instead — see below), --fast-steps (opt-in, use a faster model for per-step decisions on Groq/Gemini — see below), --story path (split a raw multi-AC user story into independent goals and run them concurrently instead of a single --goal — see below), --action-cache (opt-in, skip the upfront planning LLM call on a repeat run of the same goal by reusing a previously-saved plan — see below), --verify-clean-session and --verify (prove the generated spec stands on its own and that its assertions actually catch failures — see below), --report path (also write a machine-readable JSON summary — see below).

A successful run writes a real, human-readable Playwright .spec.ts file containing every confirmed-working step — re-runnable any time via npx playwright test. The run itself is not deterministic (the same goal against the same page can take a different path next time); the generated spec is the frozen, repeatable artifact.

A failed assertion is reported as a real finding about the app (with a screenshot, DOM snapshot, and a root-cause hypothesis), clearly separated from a tooling hiccup (an unparseable LLM response, a stuck/repeating agent) — the two are never conflated.

Exit codes are CI-friendly: five46 test/five46 api exit 0 only when the goal was actually reached, and non-zero for anything else (a failed assertion, a stuck/looping run, a missing API key, ...) — so five46 test <url> --goal "..." || exit 1 in a CI script works as expected.

Proving the generated spec is real

Two opt-in checks run after a successful run (five46 test and five46 api), each costing one extra execution of the generated spec and no extra LLM calls:

five46 test http://localhost:3000 --goal "..." --verify-clean-session --verify
  • --verify-clean-session re-runs the just-written spec standalone, with no reused --storage-state, cookies or auth headers, and fails the run if the spec doesn't pass on its own. Neither generator ever writes session state into the spec file, so this is a real check of whether the artifact works from a cold start.
  • --verify is a mutation-style negative control. It flips every value-specific assertion's expected value in a scratch copy of the spec, re-runs that copy, and requires it to fail. That proves the assertions are load-bearing rather than just present. Presence-only assertions (assert_visible, assert_json_path_exists) and assert_page_text_absent can't be mutated and are skipped, with a clear "nothing to verify" message.

Either check failing makes the command exit non-zero, exactly like a failed goal. Separately, every run prints a warning when the generated spec's passing assertions are mostly or entirely presence-only (confirming something loaded, not that it has the right value). No flag is needed for that warning.

Machine-readable run reports

five46 test http://localhost:3000 --story criteria.md --report five46-report.json

--report <path> (on five46 test and five46 api) also writes a JSON summary of the run, for CI integrations that need more than an exit code, such as a pull-request comment or a dashboard upload. A single --goal run is reported as one scenario, so --goal and --story produce the same shape:

{
  "schemaVersion": 1,
  "tool": "five46",
  "toolVersion": "0.5.0",
  "command": "test",
  "target": "http://localhost:3000",
  "passed": false,
  "generatedAt": "2026-10-19T12:00:00.000Z",
  "scenarios": [
    {
      "index": 1,
      "goal": "log in with valid credentials and confirm the dashboard loads",
      "outcome": "goal-reached",
      "passed": true,
      "specPath": "five46-agent-abc123.spec.ts",
      "artifactDir": "five46-agent-abc123-ac1",
      "details": "..."
    }
  ]
}

outcome is a run outcome (goal-reached, assertion-failed, stuck-repeating, ...) or tooling-error when five46 itself failed before reaching one (the job log has the reason). passed follows the exit-code rule exactly. details is the same secret-redacted report printed to the terminal, including the root-cause hypothesis. The report is skipped (and the job log explains why) if five46 can't start at all, for example with no API key, so treat a missing file as a failure. --report isn't supported together with --repeat. schemaVersion changes only on a breaking change; new fields can be added without one.

Testing behind a login

Capture a session once, reuse it across runs:

export FIVE46_LOGIN_USERNAME=...
export FIVE46_LOGIN_PASSWORD=...

five46 login https://your-app.example.com/login --goal "log in" --out session.json
five46 test https://your-app.example.com/dashboard --goal "..." --storage-state session.json

Your username/password are never sent to the LLM — the model only ever sees placeholder tokens; the real values are substituted locally at the point Playwright actually types them. session.json is itself a live bearer credential — treat it like one: don't commit it (it's written with user-only file permissions).

API/backend testing

No browser involved — the same agentic engine drives real HTTP requests toward a goal instead, and writes a real, standalone node:test script (plain node:test + node:assert + native fetch, no Playwright needed):

five46 api https://api.your-app.example.com --goal "create a user, then fetch it back and confirm the name matches"

Read-only (GET/HEAD/OPTIONS) by default. Add --allow-writes to unlock POST/PUT/PATCH, and --allow-deletes to separately unlock DELETE. Requests are restricted to the target's own origin unless you name another one via repeatable --allow-host <host>.

Listing past runs

five46 list          # current directory
five46 list ./tests  # or any other directory
five46 list --project checkout   # only runs tagged with this project

Lists previously generated five46-agent-*.spec.ts/five46-api-*.test.mjs files with their goal and outcome, most recent first. No separate "rerun" command — every generated file already is a real, standalone Playwright/ node:test file: npx playwright test <file> / node --test <file>.

Diffing two runs

five46 diff five46-agent-abc123.spec.ts five46-agent-def456.spec.ts

A plain line diff between any two generated (or other text) files, with the header's run-id token ignored (the outcome half of that same line is still compared). Exits 0 if identical, 1 if they differ.

Flaky-test detection

five46 test http://localhost:3000 --goal "..." --repeat 5

Runs the same goal N times (sequentially — capped at 10) and reports whether it's flaky: either the outcome differed across runs, or every run reached the goal but took a genuinely different path. Exits 0 only if every repeat succeeded with byte-identical generated output. Works the same way on five46 api.

Project management

// five46.config.json
{
  "projects": {
    "checkout": { "url": "http://localhost:3000/checkout", "storageState": "session.json" }
  }
}
five46 test --goal "..." --project checkout

A CLI flag always wins over a project default; a project only fills in what you didn't pass. --goal is never project-configurable. The LLM API key is never sourced from this file — only a provider label can be.

Video replay

five46 test http://localhost:3000 --goal "..." --record-video

Records the whole session as a real .webm (also available on five46 login). No special "replay" command — open the file in any video player.

Structured planning

On by default. One extra LLM call plans the whole goal upfront; most steps then execute directly against the real page/response with no further live decision — falling back to a normal live decision only when a step's prediction doesn't resolve cleanly. Same safety guarantees as an ordinary run (destructive-click gating, method/host allowlisting) are enforced independently at the fast path too, not skipped. This is the single biggest lever for cutting a run's wall-clock time, since LLM round-trip latency — not five46's own code — is the dominant per-run cost.

Confirming an outcome fast-paths too, not just navigating to it: a planned assert_visible step fast-paths under the same rule as clicks/fills (its target must resolve to exactly one real element, or it falls back to a live decision) — the visibility check itself still runs for real against the live page either way, nothing is assumed. assert_text/assert_value/ assert_page_text/assert_page_text_absent always make a live decision, since they'd also need to predict an exact expected text/value for a page the plan never saw, a bigger risk than predicting an element's role/name. A well-formed goal can often complete with a single LLM call total (the upfront plan) if every step, including the final confirmation, fast-paths.

five46 test http://localhost:3000 --goal "..." --no-structured-plan

--no-structured-plan opts back into the fully-adaptive loop (a live decision every single step) — works the same way on five46 api. Note: this default applies to the test/api CLI commands only — an ordinary MCP-driven goal/story call always uses the fully-adaptive loop instead, since structured planning isn't independently exposed as an MCP tool parameter. The one exception: an MCP call that supplies steps (see the MCP docs) forces structured planning on for that call, since that's what grounds it.

Cross-run action cache

five46 test http://localhost:3000 --goal "..." --action-cache

Opt-in, off by default, test/api CLI commands only (no MCP equivalent). Requires structured planning to be on (the default) — it's an alternative source for the upfront plan, not a parallel mechanism, so it has nothing to do with --no-structured-plan set. On a goal-reached run, the whole plan is saved to ~/.five46/cache.json, keyed by project/goal/URL; the next run of that same goal skips the upfront planning LLM call entirely and reuses the cached plan — worthwhile when you're re-running the same check repeatedly (e.g. in a tight edit-test loop) and want to shave off that call's latency/cost every time. Every step is still resolved live against the real page/response before it runs, exactly like an ordinary structured-plan run — a stale cache (the page genuinely changed since it was written) is detected automatically and just falls back to a fresh live plan call, never a wrong guess. Disabled automatically during --repeat, since flakiness detection needs a genuinely fresh live decision on every repeat.

Fast per-step decisions

five46 test http://localhost:3000 --goal "..." --fast-steps

Opt-in — off by default. On Groq and Gemini, swaps in a genuinely faster model tier (openai/gpt-oss-20b, gemini-flash-lite-latest) for the high-frequency per-step action-decision call only; the one-time upfront plan (and the root-cause hypothesis call, if triggered) always use your configured model, never the fast one. On OpenAI, Anthropic, and Bedrock this flag has no effect — each is already at the fastest model tier that provider offers without risking reliability on the strict JSON-only action schema.

This is a real tradeoff, not a free win: a smaller/faster model is a genuine, currently-unquantified risk to per-step decision quality (more wrong-ref picks or retries), which is exactly why it's opt-in rather than default like structured planning. If wall-clock speed matters most to you, also consider picking Groq as your provider in the first place — see Getting a key.

Story mode

five46 test http://localhost:3000 --story user-story.txt --concurrency 3

A real user story or Jira ticket often bundles several acceptance criteria together — often independent, sometimes mutually-exclusive scenarios ("checkout succeeds with valid info," "checkout fails with an invalid coupon") that can't coexist in one linear run. --story <path> reads the raw story text, splits it into independent goals with one extra upfront LLM call, then runs each one exactly like an ordinary --goal run — same safety gating, same structured planning/--fast-steps speed path, its own fresh browser session/artifact directory, its own generated spec file — with up to --concurrency running at once (default 2 for five46 test, 3 for five46 api — a real browser session against one origin carries a heavier, more bot-like footprint than a plain HTTP request, so the browser default is lower; both are hard-capped at 5). Mutually exclusive with --goal — pass exactly one. Works the same way on five46 api.

Split into 2 scenario(s):
  AC1: log in with valid credentials and complete checkout, then confirm the order confirmation is shown
  AC2: attempt checkout with no shipping details entered, then confirm a validation error is shown

=== Story mode summary ===
AC1: PASS — log in with valid credentials and complete checkout, then confirm the order confirmation is shown
AC2: PASS — attempt checkout with no shipping details entered, then confirm a validation error is shown

2/2 acceptance criteria reached goal-reached.

Exit code is all-or-nothing, matching --repeat's own CI philosophy: 0 only if every acceptance criterion reached goal-reached. Splitting is itself an LLM call and could occasionally group scenarios in an unintended way — a malformed or unusable split response degrades safely to treating the whole story as a single goal, never worse than not using --story at all.