diff --git a/CHANGELOG.md b/CHANGELOG.md index a891b37..0dfd7a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,8 +26,8 @@ a CI job fails when they disagree. `GITHUB_EVENT_NAME` is `pull_request` (or with `--channel pr`). The pull request's own job still names its build by hash. Builds uploaded before this count as merged builds, and the same bytes republished from main come off the - channel. The `/api/v1` artifact document does not show the channel yet; - `GET /api/artifacts` does. + channel. The `/api/v1` artifact document shows it as `channel: pr|main` + since #96. - **`FLEET_PEER_TOKENS`: the brain switcher can read a peer with tokens on.** The peers proxy sent no token, so a peer running `FLEET_AUTH=tokens` answered 401 through it. Each peer's URL now maps to a `read` token that @@ -95,6 +95,52 @@ a CI job fails when they disagree. pull request, docs-only ones included, when a marked number disagrees with the shipped entries in `collector/src/workloads/catalog.ts`. The collector's suite checks that reading against the real `CATALOG`. ([#89](https://github.com/addisdev/fleet-runner/pull/89)) +- **Findings are in the control API.** `/api/v1` had 22 routes and none was + findings, so a coding agent could run a job but not read the bug list night + QA wrote for it. `GET /api/v1/findings` (filter by `app`, `state`, + `verdict`, `check`, `since`, which takes `24h` as well as ISO 8601), + `GET /api/v1/findings/{id}` (the steps in words, the evidence as links, the + replay flow inline up to 16 KiB, the regression test, the verdict history) + and `POST /api/v1/findings/{id}/verdict` (scope `run`, audited). The legacy + `/api/findings` routes call the same functions and gained the same filters + and fields. ([#96](https://github.com/addisdev/fleet-runner/pull/96)) +- **`fleetr findings`, `fleetr finding`, `fleetr verdict`, and three MCP + tools.** "What did last night find in GreenFolio, and give me the Maestro + flow" is now one call: `fleetr findings --app greenfolio --since 24h`, then + `fleetr finding ID --flow repro.yaml`; `fleet_findings`, `fleet_finding` + (flow text inline) and `fleet_verdict` (not read-only, scope `run`) do the + same over MCP. `fleetr findings --since 24h --json` is the input a scheduled + agent session can start from; its shape is documented in docs/agents.md. + The skill has a section on reading a finding: the replay count first, + `agent_mistake` is not a bug, and never `fixed` without a fix merged. ([#96](https://github.com/addisdev/fleet-runner/pull/96)) +- **A fixed finding becomes a regression test.** A new verdict, `fixed`, comes + only after `real` (a 409 otherwise). Marking a finding real copies its replay + flow into a new `regressions` table as a candidate; fixed makes it active; + duplicate, not a bug or agent's mistake retire it. A nightly `ui-test` with + `params.regressions: ""` pulls the app's active flows from the + collector at run time and runs each after its own flow, so a finding marked + fixed today is in tonight's run without anyone editing a schedule. For a + crash or ANR finding on Android the executor reads `logcat -b crash` and + `am_anr` around the flow; a new crash of the package is `reproduced`, fails + the row and reopens the finding (verdict cleared, seen once more, the build + added, a `regressed on build ...` line in its new verdict history). Every + other check can only be asserted as "replays without error", and the docs + and each run's detail say so; a failing flow fails the row but reopens + nothing. GitHub issues are only ever created, never read back, so closing + one does not mark a finding fixed. The dashboard has a **Fixed** button + (f) and a regression panel. A database opened by this release + rebuilds the findings table once to allow `fixed` and backfills one history + row per existing verdict. Not yet run on a real nightly: covered by the + suite against fake crash readings and a fake Maestro. ([#96](https://github.com/addisdev/fleet-runner/pull/96)) +- **The Findings page reads and judges through `/api/v1`.** Its list, detail + and verdict calls go through the generated client, so its types are the API's + and a dashboard verdict is the same call as `fleetr verdict`. `GET + /api/findings`, `GET /api/findings/{id}` and `POST + /api/findings/{id}/verdict` now carry `Deprecation` and a successor `Link`; + `/api/findings/precision`, `/digest` and `/issues` have no v1 counterpart and + do not. The v1 list carries `facets` (every app and check, and open/triaged + counts) like the device and job lists, and the v1 verdict keeps the + `FLEET_DASH_TOKEN` speed bump the older route had. Part of #83. ([#96](https://github.com/addisdev/fleet-runner/pull/96)) ### Changed @@ -195,6 +241,10 @@ a CI job fails when they disagree. ### Fixed +- **The OpenAPI document named the wrong status for two enqueue refusals.** + `POST /api/v1/jobs` listed `no_matching_devices` and `no_build` under 422, + but `queue.ts` refuses both with 400. The 400 and 422 entries now name the + codes each really carries. ([#96](https://github.com/addisdev/fleet-runner/pull/96)) - **The older job routes skipped two auth checks.** With `FLEET_AUTH=tokens`, `POST /api/jobs/:id/cancel` and `/retry` let any `run` token stop or re-run anyone's job, and `POST /jobs` and `POST /api/jobs` accepted the `shell` diff --git a/collector/dash/src/api.ts b/collector/dash/src/api.ts index 35cd59b..7e13e13 100644 --- a/collector/dash/src/api.ts +++ b/collector/dash/src/api.ts @@ -230,69 +230,12 @@ export type Health = { // --- night-QA findings (mirrors src/findings.ts) --- -export type FindingVerdict = "real" | "duplicate" | "not_a_bug" | "agent_mistake"; - -export type FindingIssue = { - state: "dry_run" | "pending" | "filed" | "failed" | "capped" | "no_repo"; - /** True means nothing was sent to GitHub. */ - dry_run: boolean; - repo: string | null; - title: string; - body: string; - labels: string[]; - url: string | null; - number: number | null; - detail: string; - at: string; -}; - -export type Finding = { - id: number; - fingerprint: string; - app: string; - build: string; - platform: string; - device_id: string; - job_id: string; - last_job_id: string | null; - mission_id: string; - /** An open string, like Device.platform: the workload gains checks before the dashboard learns their names. */ - check: string; - severity: "high" | "medium" | "low"; - title: string; - detail: string; - screen: string; - screen_name: string; - steps: string[]; - replay: { kind: string; sha256: string | null; attempts: number; reproduced: number } | null; - /** "reproduced 2/2", "flaky 1/2", "crash log". */ - replay_label: string; - artifacts: { shot?: string; sheet?: string; trajectory?: string; log?: string; replay?: string }; - first_seen: string | null; - last_seen: string | null; - seen_count: number; - builds_seen: string[]; - verdict: FindingVerdict | null; - verdict_note: string | null; - verdict_at: string | null; - duplicate_of: number | null; - issue: FindingIssue | null; - status: "open" | "triaged"; -}; - -export type FindingDetail = Finding & { - duplicates: { id: number; title: string; seen_count: number }[]; - duplicate_of_title: string | null; -}; - -export type FindingList = { - findings: Finding[]; - counts: { open: number; triaged: number; total: number }; - /** Facets over every finding, not the filtered set. */ - apps: string[]; - checks: string[]; - limit: number; -}; +// Findings, one finding and the verdict come from /api/v1, so their types are +// the generated client's. The precision table is not in /api/v1 and keeps its +// own type below. +export type { Finding, FindingDetail, FindingList } from "./client.js"; +import type { Finding as V1Finding } from "./client.js"; +export type FindingVerdict = NonNullable; export type FindingPrecision = { floor: number; diff --git a/collector/dash/src/keys.ts b/collector/dash/src/keys.ts index 8d4a966..34f2809 100644 --- a/collector/dash/src/keys.ts +++ b/collector/dash/src/keys.ts @@ -21,7 +21,7 @@ const GOTO: Record = { export const SHORTCUTS = [ ...Object.entries(GOTO).map(([k, [, label]]) => ({ keys: `g ${k}`, does: label })), { keys: "/", does: "Focus the search box on this screen" }, - { keys: "r d n a", does: "On a finding: Real, Duplicate, Not a bug, Agent's mistake" }, + { keys: "r d n a f", does: "On a finding: Real, Duplicate, Not a bug, Agent's mistake, Fixed (after Real)" }, { keys: "?", does: "Show this help" }, { keys: "Esc", does: "Close help, or leave the search box" }, ]; diff --git a/collector/dash/src/pages/Findings.tsx b/collector/dash/src/pages/Findings.tsx index edddbaa..d14d75d 100644 --- a/collector/dash/src/pages/Findings.tsx +++ b/collector/dash/src/pages/Findings.tsx @@ -10,8 +10,9 @@ // is switched off for that app (the precision table at the top), so a wrong // verdict costs tomorrow night something, and the buttons say what they mean. import { useEffect, useRef, useState } from "preact/hooks"; -import { useApi, type Finding, type FindingDetail as Detail, type FindingList, type FindingPrecision, type FindingVerdict } from "../api.js"; -import { mutate, useMutation } from "../mutate.js"; +import { useApi, useLoad, type Finding, type FindingDetail as Detail, type FindingPrecision, type FindingVerdict } from "../api.js"; +import { change, fleet, read } from "../client.js"; +import { useMutation } from "../mutate.js"; import { useDeviceNames } from "../names.js"; import { navigate, useQuery } from "../router.js"; import { @@ -28,6 +29,7 @@ export const VERDICT_LABEL: Record = { duplicate: "Duplicate", not_a_bug: "Not a bug", agent_mistake: "Agent's mistake", + fixed: "Fixed", }; const VERDICT_PILL: Record = { @@ -35,6 +37,7 @@ const VERDICT_PILL: Record = { duplicate: "queued", not_a_bug: "cancelled", agent_mistake: "cancelled", + fixed: "done", }; /** reproduced N/N is solid, flaky is a warning, a crash log is its own evidence. */ @@ -166,10 +169,12 @@ export function Findings() { const app = q.get("app") ?? ""; const check = q.get("check") ?? ""; const status = (STATUSES as readonly string[]).includes(q.get("status") ?? "") ? (q.get("status") as string) : "open"; - const params = new URLSearchParams({ status }); - if (app) params.set("app", app); - if (check) params.set("check", check); - const state = useApi(`/api/findings?${params}`, ["finding"], 60_000); + const state = useLoad( + `findings ${status} ${app} ${check}`, + () => read(fleet.listFindings({ state: status as "open" | "triaged" | "all", app: app || undefined, check: check || undefined, per_page: 100 })), + ["finding"], + 60_000, + ); return ( <> @@ -178,7 +183,7 @@ export function Findings() { {(d) => ( {STATUSES.map((s) => ( @@ -191,19 +196,19 @@ export function Findings() { } > setQuery({ app: null, check: null })}> - setQuery({ check: v })} /> + setQuery({ check: v })} /> - {d.findings.length === 0 ? ( + {d.items.length === 0 ? (

- {d.counts.total === 0 + {d.facets.counts.total === 0 ? "Nothing found yet. Night QA posts here after it has replayed a finding on a clean install." : status === "open" ? "Nothing waiting for a verdict." : "No findings match."}

) : ( - + )} )} @@ -231,7 +236,9 @@ const typing = (t: EventTarget | null) => { /** * The four buttons, the note, and the duplicate picker. * - * r / d / n / a do the same as the buttons. They are ignored straight after a + * r / d / n / a / f do the same as the buttons. Fixed is offered only once a + * finding is real (the collector refuses it otherwise): it makes the finding's + * replay flow a regression test in the app's nightly. They are ignored straight after a * `g`, because "g r" is already "go to Results" and a shortcut that filed a * verdict on the way out of the page would be a nasty surprise. */ @@ -240,19 +247,21 @@ function VerdictBox({ f, onDone }: { f: Detail; onDone: () => void }) { const [picking, setPicking] = useState(false); const [dupOf, setDupOf] = useState(""); const pickRef = useRef(null); - const others = useApi(picking ? `/api/findings?app=${encodeURIComponent(f.app)}&status=all&limit=200` : null); + const others = useLoad(picking ? `findings of ${f.app}` : null, () => read(fleet.listFindings({ app: f.app, state: "all", per_page: 200 }))); const [pending, setPending] = useState(null); + // Fixed follows real, or a regression test that brought the finding back. + const canFix = f.verdict === "real" || f.verdict === "fixed" || f.regression?.state === "active"; const send = useMutation(async () => { const body: Record = { verdict: pending, note: note.trim() || null }; if (pending === "duplicate") body.duplicate_of = Number(dupOf); - const r = await mutate("POST", `/api/findings/${f.id}/verdict`, body); + const r = await change(fleet.setFindingVerdict(String(f.id), body as Parameters[1])); setPicking(false); onDone(); return r; }); const reopen = useMutation(async () => { - const r = await mutate("POST", `/api/findings/${f.id}/verdict`, { verdict: null }); + const r = await change(fleet.setFindingVerdict(String(f.id), { verdict: null })); onDone(); return r; }); @@ -288,7 +297,8 @@ function VerdictBox({ f, onDone }: { f: Detail; onDone: () => void }) { afterG = false; return; } - const v = ({ r: "real", d: "duplicate", n: "not_a_bug", a: "agent_mistake" } as Record)[e.key]; + const v = ({ r: "real", d: "duplicate", n: "not_a_bug", a: "agent_mistake", f: "fixed" } as Record)[e.key]; + if (v === "fixed" && !canFix) return; if (v) { e.preventDefault(); judge(v); @@ -299,9 +309,9 @@ function VerdictBox({ f, onDone }: { f: Detail; onDone: () => void }) { removeEventListener("keydown", onKey); clearTimeout(timer); }; - }, [f.id, note]); + }, [f.id, note, canFix]); - const candidates = (others.data?.findings ?? []).filter((o) => o.id !== f.id); + const candidates = (others.data?.items ?? []).filter((o) => o.id !== f.id); return ( void }) { {VERDICT_LABEL[f.verdict]} · {clock(f.verdict_at)}{" "} - ) : keys: r d n a} + ) : keys: r d n a f} >
{(Object.keys(VERDICT_LABEL) as FindingVerdict[]).map((v) => ( @@ -318,12 +328,14 @@ function VerdictBox({ f, onDone }: { f: Detail; onDone: () => void }) { key={v} tone={f.verdict === v ? "primary" : undefined} busy={send.busy && pending === v} + disabled={v === "fixed" && !canFix} onClick={() => judge(v)} title={{ real: "A bug in the app. Counts towards this check's precision.", duplicate: "The same problem as another finding. Left out of precision.", not_a_bug: "The app is fine; the check was wrong. Counts against this check.", agent_mistake: "The model did something silly and the app reacted correctly. Counts against this check.", + fixed: "Real, and a fix has merged. Its replay flow joins the app's nightly ui-test as a regression test. Only after Real.", }[v]} > {v === "not_a_bug" ? "n" : v[0]} {VERDICT_LABEL[v]} @@ -361,8 +373,8 @@ function VerdictBox({ f, onDone }: { f: Detail; onDone: () => void }) { /** The next finding still waiting for a verdict, so a morning's triage is a run of keystrokes. */ function NextOpen({ f }: { f: Detail }) { - const list = useApi(`/api/findings?status=open&limit=200`, ["finding"]); - const next = list.data?.findings.find((o) => o.id !== f.id); + const list = useLoad("findings open", () => read(fleet.listFindings({ state: "open", per_page: 200 })), ["finding"]); + const next = list.data?.items.find((o) => o.id !== f.id); if (!list.data) return null; return next ? ( next open: {next.title} → @@ -398,8 +410,44 @@ function IssuePanel({ issue }: { issue: NonNullable }) { ); } +const REGRESSION_LABEL: Record["state"], string> = { + candidate: "candidate — joins the nightly when marked fixed", + active: "active — in the nightly", + retired: "retired", +}; + +/** The regression test a real finding became, and the verdict history that moved it. */ +function RegressionPanel({ r, history }: { r: NonNullable; history: Detail["history"] }) { + return ( + {REGRESSION_LABEL[r.state]}} + > +

+ {r.kind} flow · {r.asserts === "no_crash" ? "asserts the app does not crash during it" : "asserts only that it replays without error"} + {" "}· run {r.runs}× + {r.last_outcome && <> · last {r.last_outcome}{r.last_run_at ? ` ${agoFrom(r.last_run_at)}` : ""}} + {r.last_job_id && <> in {r.last_job_id}} +

+ {r.last_detail &&

{r.last_detail}

} + {history.length > 0 && ( +
    + {history.map((h, i) => ( +
  • + {clock(h.at)}{" "} + {h.event === "regressed" ? "regressed" : h.verdict ? VERDICT_LABEL[h.verdict] : "reopened"} + {h.actor && · {h.actor}} + {h.note && <> — {h.note}} +
    • + ))} +
+ )} + + ); +} + export function FindingPage({ id }: { id: string }) { - const state = useApi(`/api/findings/${encodeURIComponent(id)}`, ["finding"]); + const state = useLoad(`finding ${id}`, () => read(fleet.getFinding(id)), ["finding"]); const names = useDeviceNames(); return ( @@ -494,6 +542,8 @@ export function FindingPage({ id }: { id: string }) { )} + {f.regression && } + {f.issue && } diff --git a/collector/openapi.json b/collector/openapi.json index e7a46c9..8f4eeef 100644 --- a/collector/openapi.json +++ b/collector/openapi.json @@ -45,6 +45,10 @@ "name": "artifacts", "description": "Screenshots, logs, reports, builds." }, + { + "name": "findings", + "description": "What night QA found, its verdicts, and the regression tests fixed findings became." + }, { "name": "fleet", "description": "Schedules, alerts, audit." @@ -616,7 +620,7 @@ } }, "400": { - "description": "The request was malformed or the spec is invalid.", + "description": "Malformed or refused before queueing: invalid_spec, no_matching_devices (no registered device matches the targets), no_build (no artifact for app `latest`), invalid_match, invalid_dependency, secret_in_spec, bad_request.", "content": { "application/json": { "schema": { @@ -656,7 +660,7 @@ } }, "422": { - "description": "Valid, but nothing can run it (no capable agent, no matching device, no build).", + "description": "Valid, but cannot be queued: no_capable_agent (nothing declares the workload), unresolved_reference (a template reference that does not resolve).", "content": { "application/json": { "schema": { @@ -1295,6 +1299,256 @@ } } }, + "/api/v1/findings": { + "get": { + "operationId": "listFindings", + "summary": "What night QA (the explore workload) found, newest sighting first.", + "description": "Each finding was replayed on a clean install before it was filed; `replay_label` says how often it reproduced. verdict agent_mistake means the model misread the app, not a bug in it. since=24h is the morning brief.", + "tags": [ + "findings" + ], + "x-fleet-scope": "read", + "x-fleet-read-only": true, + "x-fleet-idempotent": true, + "parameters": [ + { + "name": "app", + "in": "query", + "required": false, + "description": "Only this app (the explore app key).", + "schema": { + "type": "string" + } + }, + { + "name": "state", + "in": "query", + "required": false, + "description": "open: no verdict yet (or reopened by a regression). Default all.", + "schema": { + "type": "string", + "enum": [ + "open", + "triaged", + "all" + ] + } + }, + { + "name": "verdict", + "in": "query", + "required": false, + "description": "Comma-separated: real, duplicate, not_a_bug, agent_mistake, fixed.", + "schema": { + "type": "string" + } + }, + { + "name": "check", + "in": "query", + "required": false, + "description": "crash, anr, frozen, blank, left_app, a11y, visual, goal, dead_control.", + "schema": { + "type": "string" + } + }, + { + "name": "since", + "in": "query", + "required": false, + "description": "Last seen at or after: ISO 8601, or a span back from now such as 90m, 24h, 7d.", + "schema": { + "type": "string" + } + }, + { + "name": "page", + "in": "query", + "required": false, + "description": "1-based page.", + "schema": { + "type": "integer", + "default": 1, + "minimum": 1 + } + }, + { + "name": "per_page", + "in": "query", + "required": false, + "description": "Items per page.", + "schema": { + "type": "integer", + "minimum": 1, + "maximum": 500 + } + } + ], + "responses": { + "200": { + "description": "Findings, and the facets a filter menu is built from.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/FindingList" + } + } + } + }, + "400": { + "description": "The request was malformed or the spec is invalid.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/Error" + } + } + } + } + } + } + }, + "/api/v1/findings/{id}": { + "get": { + "operationId": "getFinding", + "summary": "One finding: the steps in words, its evidence as links, the replay flow (inline when small), its regression test and verdict history.", + "tags": [ + "findings" + ], + "x-fleet-scope": "read", + "x-fleet-read-only": true, + "x-fleet-idempotent": true, + "parameters": [ + { + "name": "id", + "in": "path", + "required": true, + "description": "The finding's id.", + "schema": { + "type": "integer" + } + } + ], + "responses": { + "200": { + "description": "The finding.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/FindingDetail" + } + } + } + }, + "404": { + "description": "No such thing.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/Error" + } + } + } + } + } + } + }, + "/api/v1/findings/{id}/verdict": { + "post": { + "operationId": "setFindingVerdict", + "summary": "Judge a finding: real, duplicate, not_a_bug, agent_mistake, or fixed (only after real). null reopens.", + "description": "real keeps the finding's replay flow as a candidate regression test; fixed makes it active, so the app's nightly ui-test (with params.regressions) runs it. duplicate, not_a_bug and agent_mistake retire it. Mark fixed only once a fix has merged. Each verdict counts towards the check's precision, which decides what night QA runs tomorrow.", + "tags": [ + "findings" + ], + "x-fleet-scope": "run", + "x-fleet-idempotent": true, + "parameters": [ + { + "name": "id", + "in": "path", + "required": true, + "description": "The finding's id.", + "schema": { + "type": "integer" + } + } + ], + "requestBody": { + "required": true, + "description": "The verdict and an optional note.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/FindingVerdictInput" + } + } + } + }, + "responses": { + "200": { + "description": "Recorded.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/FindingVerdictResult" + } + } + } + }, + "400": { + "description": "The request was malformed or the spec is invalid.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/Error" + } + } + } + }, + "401": { + "description": "A token is required (FLEET_AUTH=tokens).", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/Error" + } + } + } + }, + "403": { + "description": "The token lacks the scope, or the job is someone else's.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/Error" + } + } + } + }, + "404": { + "description": "No such thing.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/Error" + } + } + } + }, + "409": { + "description": "fixed on a finding that is not real (conflict).", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/Error" + } + } + } + } + } + } + }, "/api/v1/schedules": { "get": { "operationId": "listSchedules", @@ -1521,7 +1775,7 @@ "post": { "operationId": "uploadArtifact", "summary": "Upload bytes (application/octet-stream). Answers with the sha256 and the detected media type.", - "description": "Headers: x-artifact-name, x-artifact-media-type (optional), x-artifact-app, x-artifact-build.", + "description": "Headers: x-artifact-name, x-artifact-media-type (optional), x-artifact-app, x-artifact-build, x-artifact-channel (optional: `pr` for a pull request's build, which `\"sha256\": \"latest\"` never resolves to; absent means a merged build).", "tags": [ "protocol" ], @@ -1533,39 +1787,98 @@ } } }, - "/artifacts/{sha256}": { + "/findings": { + "post": { + "operationId": "postFinding", + "summary": "The explore workload filing a finding (an ExploreFinding). Upserts on the fingerprint.", + "tags": [ + "protocol" + ], + "x-fleet-scope": "protocol", + "responses": { + "200": { + "description": "Seen again." + }, + "201": { + "description": "New." + } + } + } + }, + "/findings/regressions": { "get": { - "operationId": "downloadArtifact", - "summary": "The bytes as application/octet-stream, with Range support. Runners fetch models and builds here.", + "operationId": "listRegressionFlows", + "summary": "A nightly ui-test pulling an app's regression flows (?app= &state=active &platform=).", + "tags": [ + "protocol" + ], + "x-fleet-scope": "protocol", + "responses": { + "200": { + "description": "{regressions: [...]}, each with its flow text." + } + } + } + }, + "/findings/{id}/regression": { + "post": { + "operationId": "postRegressionRun", + "summary": "An executor reporting one regression run: {job_id, device_id, build, outcome: passed|reproduced|failed|skipped, detail}. reproduced reopens the finding.", "tags": [ "protocol" ], "x-fleet-scope": "protocol", "parameters": [ { - "name": "sha256", + "name": "id", "in": "path", "required": true, - "description": "The artifact's sha256 (64 hex).", + "description": "The finding's id.", "schema": { - "type": "string" + "type": "integer" } } ], "responses": { "200": { - "description": "The bytes.", - "content": { - "application/octet-stream": {} - } + "description": "Recorded." } } } - } - }, - "components": { - "schemas": { - "Alert": { + }, + "/artifacts/{sha256}": { + "get": { + "operationId": "downloadArtifact", + "summary": "The bytes as application/octet-stream, with Range support. Runners fetch models and builds here.", + "tags": [ + "protocol" + ], + "x-fleet-scope": "protocol", + "parameters": [ + { + "name": "sha256", + "in": "path", + "required": true, + "description": "The artifact's sha256 (64 hex).", + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "The bytes.", + "content": { + "application/octet-stream": {} + } + } + } + } + } + }, + "components": { + "schemas": { + "Alert": { "type": "object", "properties": { "id": { @@ -1689,6 +2002,13 @@ "pinned": { "type": "boolean" }, + "channel": { + "enum": [ + "main", + "pr" + ], + "description": "pr: uploaded from a pull request (x-artifact-channel: pr); `\"sha256\": \"latest\"` never resolves to it. main: everything else." + }, "in_store": { "type": "boolean" }, @@ -1701,6 +2021,7 @@ "name", "size", "media_type", + "channel", "in_store", "url" ] @@ -2493,6 +2814,805 @@ "error" ] }, + "Finding": { + "type": "object", + "properties": { + "id": { + "type": "integer" + }, + "fingerprint": { + "type": "string" + }, + "app": { + "type": "string", + "description": "The explore app key the finding was filed under." + }, + "build": { + "type": "string", + "description": "The build it was first seen on." + }, + "platform": { + "type": "string" + }, + "device_id": { + "type": "string" + }, + "job_id": { + "type": "string", + "description": "The job that first found it." + }, + "last_job_id": { + "type": [ + "string", + "null" + ] + }, + "mission_id": { + "type": "string" + }, + "check": { + "type": "string", + "description": "crash, anr, frozen, blank, left_app, a11y, visual, goal, dead_control." + }, + "severity": { + "enum": [ + "high", + "medium", + "low" + ] + }, + "title": { + "type": "string" + }, + "detail": { + "type": "string" + }, + "screen": { + "type": "string" + }, + "screen_name": { + "type": "string" + }, + "steps": { + "type": "array", + "items": { + "type": "string" + }, + "description": "What to do to see it, in words, one step per entry." + }, + "replay": { + "oneOf": [ + { + "type": "null" + }, + { + "type": "object", + "properties": { + "kind": { + "enum": [ + "maestro", + "tvloop", + "none" + ] + }, + "sha256": { + "type": [ + "string", + "null" + ] + }, + "attempts": { + "type": "integer" + }, + "reproduced": { + "type": "integer" + } + }, + "required": [ + "kind", + "sha256", + "attempts", + "reproduced" + ] + } + ], + "description": "The replay on a clean install: how many of `attempts` reproduced it. null only for a crash, whose crash log is the evidence." + }, + "replay_label": { + "type": "string", + "description": "reproduced 2/2, flaky 1/2, crash log." + }, + "artifacts": { + "type": "object", + "properties": { + "shot": { + "type": "string" + }, + "sheet": { + "type": "string" + }, + "trajectory": { + "type": "string" + }, + "log": { + "type": "string" + }, + "replay": { + "type": "string" + } + }, + "required": [], + "description": "Artifact sha256s by role: shot, sheet, trajectory, log, replay. Absent roles were not posted." + }, + "first_seen": { + "type": [ + "string", + "null" + ] + }, + "last_seen": { + "type": [ + "string", + "null" + ] + }, + "seen_count": { + "type": "integer" + }, + "builds_seen": { + "type": "array", + "items": { + "type": "string" + } + }, + "verdict": { + "enum": [ + "real", + "duplicate", + "not_a_bug", + "agent_mistake", + "fixed", + null + ], + "description": "null: nobody has judged it, or a regression test reopened it. agent_mistake is NOT a bug in the app." + }, + "verdict_note": { + "type": [ + "string", + "null" + ] + }, + "verdict_at": { + "type": [ + "string", + "null" + ] + }, + "duplicate_of": { + "type": [ + "integer", + "null" + ] + }, + "issue": { + "oneOf": [ + { + "type": "null" + }, + { + "type": "object", + "properties": { + "state": { + "enum": [ + "dry_run", + "pending", + "filed", + "failed", + "capped", + "no_repo" + ] + }, + "dry_run": { + "type": "boolean", + "description": "True means nothing was sent to GitHub." + }, + "repo": { + "type": [ + "string", + "null" + ] + }, + "title": { + "type": "string" + }, + "body": { + "type": "string" + }, + "labels": { + "type": "array", + "items": { + "type": "string" + } + }, + "url": { + "type": [ + "string", + "null" + ] + }, + "number": { + "type": [ + "integer", + "null" + ] + }, + "detail": { + "type": "string" + }, + "at": { + "type": "string" + } + }, + "required": [ + "state", + "dry_run", + "repo", + "title", + "body", + "labels", + "url", + "number", + "detail", + "at" + ] + } + ], + "description": "The GitHub issue it would file, or did. Fleet Runner only ever creates issues; it never reads them back." + }, + "status": { + "enum": [ + "open", + "triaged" + ] + }, + "regression": { + "oneOf": [ + { + "type": "null" + }, + { + "type": "object", + "properties": { + "state": { + "enum": [ + "candidate", + "active", + "retired" + ] + }, + "kind": { + "enum": [ + "maestro", + "tvloop" + ] + }, + "last_outcome": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "state", + "kind", + "last_outcome" + ] + } + ], + "description": "The regression test it became: candidate once real, active (in the nightly) once fixed." + } + }, + "required": [ + "id", + "fingerprint", + "app", + "build", + "platform", + "device_id", + "job_id", + "mission_id", + "check", + "severity", + "title", + "detail", + "screen_name", + "steps", + "replay", + "replay_label", + "artifacts", + "seen_count", + "builds_seen", + "verdict", + "status", + "first_seen", + "last_seen", + "regression" + ] + }, + "FindingDetail": { + "allOf": [ + { + "$ref": "#/components/schemas/Finding" + }, + { + "type": "object", + "properties": { + "duplicates": { + "type": "array", + "items": { + "type": "object", + "properties": { + "id": { + "type": "integer" + }, + "title": { + "type": "string" + }, + "seen_count": { + "type": "integer" + } + }, + "required": [ + "id", + "title", + "seen_count" + ] + } + }, + "duplicate_of_title": { + "type": [ + "string", + "null" + ] + }, + "counts": { + "type": "object", + "properties": { + "seen": { + "type": "integer" + }, + "builds": { + "type": "integer" + }, + "replay_attempts": { + "type": [ + "integer", + "null" + ] + }, + "replay_reproduced": { + "type": [ + "integer", + "null" + ] + }, + "regression_runs": { + "type": "integer" + } + }, + "required": [ + "seen", + "builds", + "replay_attempts", + "replay_reproduced", + "regression_runs" + ] + }, + "evidence": { + "type": "array", + "items": { + "type": "object", + "properties": { + "role": { + "enum": [ + "shot", + "sheet", + "trajectory", + "log", + "replay" + ] + }, + "label": { + "type": "string" + }, + "sha256": { + "type": "string" + }, + "url": { + "type": "string", + "description": "The bytes, relative to the collector." + }, + "in_store": { + "type": "boolean" + } + }, + "required": [ + "role", + "label", + "sha256", + "url", + "in_store" + ] + } + }, + "replay_flow": { + "oneOf": [ + { + "type": "null" + }, + { + "type": "object", + "properties": { + "kind": { + "type": "string" + }, + "sha256": { + "type": "string" + }, + "url": { + "type": "string" + }, + "in_store": { + "type": "boolean" + }, + "text": { + "type": [ + "string", + "null" + ], + "description": "The flow itself when it is at most 16 KiB; otherwise fetch url." + }, + "truncated": { + "type": "boolean", + "description": "True when the flow was too big to inline." + } + }, + "required": [ + "kind", + "sha256", + "url", + "in_store", + "text", + "truncated" + ] + } + ], + "description": "The Maestro (or tvloop) flow that replays the finding's steps." + }, + "regression": { + "oneOf": [ + { + "type": "null" + }, + { + "type": "object", + "properties": { + "finding_id": { + "type": "integer" + }, + "app": { + "type": "string" + }, + "platform": { + "type": "string" + }, + "check": { + "type": "string" + }, + "kind": { + "enum": [ + "maestro", + "tvloop" + ] + }, + "state": { + "enum": [ + "candidate", + "active", + "retired" + ] + }, + "asserts": { + "enum": [ + "no_crash", + "replays" + ], + "description": "no_crash for crash and anr findings; replays (without error) for every other check." + }, + "flow": { + "type": "string", + "description": "The flow text the nightly runs, copied when the finding was marked real." + }, + "flow_sha256": { + "type": [ + "string", + "null" + ] + }, + "created_at": { + "type": [ + "string", + "null" + ] + }, + "activated_at": { + "type": [ + "string", + "null" + ] + }, + "retired_at": { + "type": [ + "string", + "null" + ] + }, + "runs": { + "type": "integer" + }, + "last_run_at": { + "type": [ + "string", + "null" + ] + }, + "last_job_id": { + "type": [ + "string", + "null" + ] + }, + "last_outcome": { + "enum": [ + "passed", + "reproduced", + "failed", + "skipped", + null + ] + }, + "last_detail": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "finding_id", + "state", + "kind", + "asserts", + "flow", + "runs" + ] + } + ] + }, + "history": { + "type": "array", + "items": { + "type": "object", + "properties": { + "at": { + "type": [ + "string", + "null" + ] + }, + "event": { + "enum": [ + "verdict", + "regressed" + ] + }, + "verdict": { + "enum": [ + "real", + "duplicate", + "not_a_bug", + "agent_mistake", + "fixed", + null + ] + }, + "note": { + "type": [ + "string", + "null" + ] + }, + "actor": { + "type": [ + "string", + "null" + ] + }, + "source": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "at", + "event", + "verdict", + "note" + ] + } + }, + "links": { + "type": "object", + "properties": { + "self": { + "type": "string" + }, + "verdict": { + "type": "string" + }, + "dashboard": { + "type": "string" + } + }, + "required": [ + "self", + "verdict", + "dashboard" + ] + } + }, + "required": [ + "duplicates", + "duplicate_of_title", + "counts", + "evidence", + "replay_flow", + "regression", + "history", + "links" + ] + } + ] + }, + "FindingList": { + "type": "object", + "properties": { + "items": { + "type": "array", + "items": { + "$ref": "#/components/schemas/Finding" + } + }, + "page": { + "type": "integer" + }, + "per_page": { + "type": "integer" + }, + "total": { + "type": "integer" + }, + "pages": { + "type": "integer" + }, + "facets": { + "type": "object", + "properties": { + "apps": { + "type": "array", + "items": { + "type": "string" + } + }, + "checks": { + "type": "array", + "items": { + "type": "string" + } + }, + "counts": { + "type": "object", + "properties": { + "open": { + "type": "integer" + }, + "triaged": { + "type": "integer" + }, + "total": { + "type": "integer" + } + }, + "required": [ + "open", + "triaged", + "total" + ] + } + }, + "required": [ + "apps", + "checks", + "counts" + ], + "description": "apps and checks over every finding, so a filter never empties its own menu; counts over the app, check and since filters, before state and verdict." + } + }, + "required": [ + "items", + "page", + "per_page", + "total", + "pages", + "facets" + ] + }, + "FindingVerdictInput": { + "type": "object", + "properties": { + "verdict": { + "enum": [ + "real", + "duplicate", + "not_a_bug", + "agent_mistake", + "fixed", + null + ], + "description": "fixed only after real, once the fix has merged. null reopens." + }, + "note": { + "type": [ + "string", + "null" + ], + "maxLength": 2000 + }, + "duplicate_of": { + "type": [ + "integer", + "null" + ], + "description": "Required with duplicate: the finding this one duplicates." + } + }, + "required": [ + "verdict" + ], + "additionalProperties": false + }, + "FindingVerdictResult": { + "type": "object", + "properties": { + "finding": { + "$ref": "#/components/schemas/FindingDetail" + }, + "change": { + "type": [ + "string", + "null" + ], + "description": "What the verdict did to the finding's regression test, in one sentence, or null." + } + }, + "required": [ + "finding", + "change" + ] + }, "FlakyCell": { "type": "object", "properties": { @@ -3344,7 +4464,7 @@ }, "params": { "type": "object", - "description": "Workload-specific. Deliberately open, but the fields the shipped workloads read are listed here so they are discoverable: benchmark/thermal prompt_tokens, gen_tokens, warmup_iters, measure_iters, duration_s; batch/vision/speech/embed input_sha256, max_items, max_tokens; cold-start app_id, launches, states; locale-shots locales; app-soak flow, duration_s, interval_s; a11y-audit steps, variants, min_target_pt, font_scale, tree_source; build repo, ref, kind, task, scheme; vantage urls, repeats; any host job network and network_to; a preempted job resume_from." + "description": "Workload-specific. Deliberately open, but the fields the shipped workloads read are listed here so they are discoverable: benchmark/thermal prompt_tokens, gen_tokens, warmup_iters, measure_iters, duration_s; batch/vision/speech/embed input_sha256, max_items, max_tokens; cold-start app_id, launches, states; locale-shots locales; app-soak flow, duration_s, interval_s; a11y-audit steps, variants, min_target_pt, font_scale, tree_source; build repo, ref, kind, task, scheme; vantage urls, repeats; ui-test regressions (true, or the app its findings are filed under: run its fixed findings' replay flows too); any host job network and network_to; a preempted job resume_from." }, "depends_on": { "type": "array", diff --git a/collector/schemas/job.schema.json b/collector/schemas/job.schema.json index 626c33d..09d68d6 100644 --- a/collector/schemas/job.schema.json +++ b/collector/schemas/job.schema.json @@ -89,7 +89,7 @@ }, "params": { "type": "object", - "description": "Workload-specific. Deliberately open, but the fields the shipped workloads read are listed here so they are discoverable: benchmark/thermal prompt_tokens, gen_tokens, warmup_iters, measure_iters, duration_s; batch/vision/speech/embed input_sha256, max_items, max_tokens; cold-start app_id, launches, states; locale-shots locales; app-soak flow, duration_s, interval_s; a11y-audit steps, variants, min_target_pt, font_scale, tree_source; build repo, ref, kind, task, scheme; vantage urls, repeats; any host job network and network_to; a preempted job resume_from." + "description": "Workload-specific. Deliberately open, but the fields the shipped workloads read are listed here so they are discoverable: benchmark/thermal prompt_tokens, gen_tokens, warmup_iters, measure_iters, duration_s; batch/vision/speech/embed input_sha256, max_items, max_tokens; cold-start app_id, launches, states; locale-shots locales; app-soak flow, duration_s, interval_s; a11y-audit steps, variants, min_target_pt, font_scale, tree_source; build repo, ref, kind, task, scheme; vantage urls, repeats; ui-test regressions (true, or the app its findings are filed under: run its fixed findings' replay flows too); any host job network and network_to; a preempted job resume_from." }, "depends_on": { "type": "array", diff --git a/collector/schemas/result.schema.json b/collector/schemas/result.schema.json index f1ab890..27e214e 100644 --- a/collector/schemas/result.schema.json +++ b/collector/schemas/result.schema.json @@ -220,7 +220,19 @@ "failed": { "type": "integer" }, "artifacts": { "type": "array", "items": { "type": "string" }, "description": "sha256 refs of screenshots/videos/junit uploaded to the artifact store" }, "skipped": { "type": "integer" }, - "failing_flow": { "type": "string", "description": "The first failing flow or test case, by name; the step that broke is in the row's error" } + "failing_flow": { "type": "string", "description": "The first failing flow or test case, by name; the step that broke is in the row's error" }, + "regressions": { + "type": "array", + "description": "ui-test with params.regressions: each fixed finding's replay flow run on this device. reproduced reopens the finding", + "items": { + "type": "object", + "properties": { + "finding_id": { "type": "integer" }, + "outcome": { "enum": ["passed", "reproduced", "failed", "skipped"] }, + "detail": { "type": "string" } + } + } + } } }, "shot": { diff --git a/collector/scripts/check-api-v1.ts b/collector/scripts/check-api-v1.ts index dd14411..f7c8610 100644 --- a/collector/scripts/check-api-v1.ts +++ b/collector/scripts/check-api-v1.ts @@ -201,6 +201,50 @@ console.log("results, artifacts, fleet"); check("an HTML artifact is never served inline", (page.headers.get("content-disposition") ?? "").startsWith("attachment")); await refusal("a malformed sha is 400", fleet.getArtifact("nothex"), 400, "bad_request"); } +console.log("findings"); +{ + check("an artifact says its channel", (await fleet.getArtifact(art.sha256)).channel === "main"); + const flow = `appId: com.example.contract\n---\n- launchApp\n# ${DEVICE}\n`; + const up = await fetch(`${BASE}/artifacts`, { method: "POST", headers: { "content-type": "application/octet-stream", "x-artifact-name": "replay.yaml" }, body: flow }); + const flowSha = ((await up.json()) as { sha256: string }).sha256; + const app = `contract-${DEVICE}`; + const posted = await raw("POST", "/findings", { + fingerprint: `${app}-1`, app, build: "1.0 (1)", platform: "android", device_id: DEVICE, job_id: "contract-explore", mission_id: "m", + check: "crash", severity: "high", title: "Crashes on save", detail: "FATAL EXCEPTION", screen: "s1", screen_name: "Save", + steps: ["Open the app", "Tap Save"], replay: { kind: "maestro", sha256: flowSha, attempts: 2, reproduced: 2 }, + artifacts: { shot: art.sha256, replay: flowSha }, + }); + const id = String(((await posted.json()) as { id: number }).id); + const page = await fleet.listFindings({ app, since: "24h" }); + conforms("listFindings", "FindingList", page); + check("listFindings finds it, in the list envelope", page.items.length === 1 && page.total === 1 && page.items[0].verdict === null); + check("listFindings facets name every app and count the open ones", page.facets.apps.includes(app) && page.facets.counts.open === 1, JSON.stringify(page.facets)); + { + const legacy = await raw("GET", `/api/findings/${id}`); + check("the older finding route is deprecated, with its successor", (legacy.headers.get("link") ?? "").includes(`; rel="successor-version"`)); + const precision = await raw("GET", "/api/findings/precision"); + check("precision, which has no successor, is not marked deprecated", precision.headers.get("deprecation") === null); + } + const one = await fleet.getFinding(id); + conforms("getFinding", "FindingDetail", one); + check("getFinding inlines the replay flow and links the evidence", + one.replay_flow?.text === flow && one.evidence.some((e) => e.role === "shot" && e.url.endsWith(`${art.sha256}/content`))); + await refusal("fixed before real is 409 conflict", fleet.setFindingVerdict(id, { verdict: "fixed" }), 409, "conflict"); + await refusal("an unknown verdict is 400", fleet.setFindingVerdict(id, { verdict: "meh" } as never), 400, "bad_request"); + const real = await fleet.setFindingVerdict(id, { verdict: "real", note: "contract" }); + conforms("setFindingVerdict (real)", "FindingVerdictResult", real); + check("real makes a candidate regression", real.finding.regression?.state === "candidate" && real.finding.history.length === 1); + const fixed = await fleet.setFindingVerdict(id, { verdict: "fixed" }); + check("fixed makes it active and says so", fixed.finding.regression?.state === "active" && /active/.test(fixed.change ?? "")); + conforms("listFindings (verdict=fixed)", "FindingList", await fleet.listFindings({ verdict: "fixed", app })); + await refusal("an unknown finding is 404", fleet.getFinding("999999999"), 404, "not_found"); + await refusal("a non-numeric finding id is 404", fleet.getFinding("abc"), 404, "not_found"); + await refusal("a bad since is 400", fleet.listFindings({ since: "whenever" }), 400, "bad_request"); + const audit = await fleet.listAudit({ per_page: 20 }); + check("a verdict is audited with its finding as the subject", + audit.items.some((a) => a.route === `/api/v1/findings/${id}/verdict` && a.subject === id && a.actor === "contract-check")); +} + { const devices = await fleet.listDevices({ pool: "v1check" }); conforms("listDevices", "DeviceList", devices); diff --git a/collector/scripts/check-auth.ts b/collector/scripts/check-auth.ts index 65953c0..ae1498d 100644 --- a/collector/scripts/check-auth.ts +++ b/collector/scripts/check-auth.ts @@ -143,6 +143,14 @@ try { const denied = await call("POST", "/api/v1/jobs", { token: READ, body: spec }); check("read token cannot create a job: 403 insufficient_scope run", denied.status === 403 && denied.body.error.code === "insufficient_scope" && denied.body.error.required_scope === "run", JSON.stringify(denied.body)); check("read token cannot use the legacy enqueue either", (await call("POST", "/jobs", { token: READ, body: { ...spec, schema: 1, job_id: "legacy-denied" } })).status === 403); + const verdictDenied = await call("POST", "/api/v1/findings/1/verdict", { token: READ, body: { verdict: "real" } }); + check("read token cannot judge a finding: 403 insufficient_scope run", verdictDenied.status === 403 && verdictDenied.body.error.required_scope === "run", JSON.stringify(verdictDenied.body)); + const verdictRun = await call("POST", "/api/v1/findings/999999/verdict", { token: ALICE, body: { verdict: "real" } }); + check("a run token may judge one (this one does not exist: 404)", verdictRun.status === 404, JSON.stringify(verdictRun.body)); + check("read token can list findings", (await call("GET", "/api/v1/findings", { token: READ })).status === 200); + check("an agent token can pull regression flows", (await call("GET", "/findings/regressions?app=x", { token: AGENT })).status === 200); + check("an agent token can report a regression run (none here: 404)", + (await call("POST", "/findings/999999/regression", { token: AGENT, body: { job_id: "j", device_id: "d", outcome: "passed" } })).status === 404); check("audit needs admin", (await call("GET", "/api/v1/audit", { token: ALICE })).status === 403); check("admin can read the audit log", (await call("GET", "/api/v1/audit", { token: ADMIN })).status === 200); const w = await call("GET", "/api/v1/whoami", { token: ADMIN }); diff --git a/collector/scripts/smoke.ts b/collector/scripts/smoke.ts index 0183f5e..b652a20 100644 --- a/collector/scripts/smoke.ts +++ b/collector/scripts/smoke.ts @@ -30,6 +30,7 @@ import { runEnrolChecks } from "../src/workloads/enrol/enrol.test.js"; import { runTvloopChecks } from "../src/workloads/tvloop/tvloop.test.js"; import { runColdStartChecks } from "../src/workloads/cold-start/cold-start.test.js"; import { runFindingsChecks } from "../src/findings.test.js"; +import { runRegressionChecks } from "../src/workloads/regressions.test.js"; import { referenceDigest } from "./conformance.js"; import { redact, keychainPassword } from "../src/secrets.js"; @@ -2888,6 +2889,10 @@ await runColdStartChecks(check); // armed GitHub path and the digest timer, against a temp database of its own. await runFindingsChecks(check); +// The ui-test side of regression tests: which app, a stored flow made +// runnable, the crash-buffer judgement, and what each run reports. +await runRegressionChecks(check); + // --- night-QA findings, over HTTP --- // // What the explore workload's executor will actually do: post findings, post @@ -2983,6 +2988,40 @@ await runFindingsChecks(check); JSON.stringify(mine.map((r) => r.issue.state))); check("findings: the issue body is composed for audit", mine.some((r) => r.issue.body.includes("Tap Save") && r.issue.labels.includes("night-qa"))); + + // A finding becomes a regression test, over HTTP: the flow is uploaded the + // way the workload uploads it, real keeps it, fixed activates it, the + // nightly's pull sees it, and a reproduced run reopens the finding. + const flowText = `appId: com.example.smoke\nname: "smoke ${run}"\n---\n- launchApp:\n clearState: true\n`; + const upFlow = await fetch(`${BASE}/artifacts`, { + method: "POST", headers: { "content-type": "application/octet-stream", "x-artifact-name": `explore-finding-${run}.yaml` }, body: flowText, + }); + const flowSha = ((await upFlow.json()) as { sha256: string }).sha256; + const rf = await json("POST", "/findings", finding({ + fingerprint: `${APP}-fp-reg`, check: "crash", replay: { kind: "maestro", sha256: flowSha, attempts: 2, reproduced: 2 }, + artifacts: { replay: flowSha }, + })); + const rid = rf.body?.id; + const tooSoon = await json("POST", `/api/findings/${rid}/verdict`, { verdict: "fixed" }); + check("regressions: fixed before real is 409 on the dashboard's route too", tooSoon.status === 409, JSON.stringify(tooSoon.body)); + const real = await json("POST", `/api/findings/${rid}/verdict`, { verdict: "real" }); + check("regressions: real keeps the uploaded flow as a candidate", real.body?.regression?.state === "candidate" && + real.body.regression.flow === flowText, JSON.stringify(real.body?.regression)); + const fixed = await json("POST", `/api/findings/${rid}/verdict`, { verdict: "fixed", note: "PR #1" }); + check("regressions: fixed activates it", fixed.status === 200 && fixed.body?.regression?.state === "active"); + const pulled = await json("GET", `/findings/regressions?app=${APP}`); + check("regressions: GET /findings/regressions is what the nightly pulls", + pulled.status === 200 && pulled.body?.regressions?.length === 1 && pulled.body.regressions[0].flow === flowText, JSON.stringify(pulled.body)); + check("regressions: a bad state filter is 400", (await json("GET", "/findings/regressions?state=maybe")).status === 400); + const runBody = { job_id: `smoke-ui-${run}`, device_id: DEVICE, build: "9.9 (1)", outcome: "reproduced", detail: "crashed again" }; + const rep = await json("POST", `/findings/${rid}/regression`, runBody); + check("regressions: a reproduced run reopens the finding", rep.status === 200 && rep.body?.reopened === true, JSON.stringify(rep.body)); + const after = await json("GET", `/api/findings/${rid}`); + check("regressions: reopened, counted, with the build and a history line", + after.body?.verdict === null && after.body?.builds_seen?.includes("9.9 (1)") && + after.body?.history?.at(-1)?.event === "regressed" && /regressed on build 9\.9 \(1\)/.test(after.body.history.at(-1).note), JSON.stringify(after.body?.history)); + check("regressions: a bad run body is 400", (await json("POST", `/findings/${rid}/regression`, { ...runBody, outcome: "meh" })).status === 400); + check("regressions: a run for a finding with no test is 404", (await json("POST", "/findings/999999999/regression", runBody)).status === 404); } // --- the synthetic backend's reference digest --- diff --git a/collector/src/api/findings.ts b/collector/src/api/findings.ts index 55fa2b4..750fb0c 100644 --- a/collector/src/api/findings.ts +++ b/collector/src/api/findings.ts @@ -9,6 +9,11 @@ * GET /api/findings/issues every issue composed, dry run or filed * GET /api/findings/:id one finding * POST /api/findings/:id/verdict a person's judgement (token-guarded) + * GET /findings/regressions what a nightly ui-test runs (?app= &state= &platform=) + * POST /findings/:id/regression an executor reporting one regression run + * + * /api/v1/findings (src/api/v1/routes.ts) calls the same functions in + * src/findings.ts; neither route set has logic of its own. * * ## Guard posture * @@ -17,11 +22,18 @@ * network (see README and FLEET_BIND), not the dashboard token. The verdict is * different -- it is a person's decision, and it decides which checks run * tomorrow night -- so it carries the same speed bump as cancel and ack. + * The two regression routes are the executor again: it pulls the flows its + * job should run and reports what each did, so they sit with `POST /findings` + * on the runner protocol's side of auth. */ import type { FastifyInstance } from "fastify"; import { LIMITS, buildDigest, + listRegressions, + parseRegressionRun, + parseVerdictBody, + recordRegressionRun, fileIssue, findingDetail, issueOptions, @@ -33,7 +45,7 @@ import { validateFinding, } from "../findings.js"; import { FINDINGS_REPOS, FINDINGS_REPOS_ERROR, GITHUB_ISSUES_ARMED, GITHUB_TOKEN } from "../config.js"; -import type { Verdict } from "../workloads/explore/types.js"; +import { authOf } from "../auth.js"; import { requireToken } from "./guard.js"; type Announce = (event: { type: string; [k: string]: unknown }) => void; @@ -75,7 +87,10 @@ export function registerFindings(app: FastifyInstance, announce: Announce) { app.get("/api/findings", async (req, reply) => { const q = req.query as Record; - const out = listFindings({ app: q.app, check: q.check, status: q.status, since: q.since, limit: Number(q.limit) || undefined }); + const out = listFindings({ + app: q.app, check: q.check, status: q.status, verdict: q.verdict, since: q.since, + limit: Number(q.limit) || undefined, offset: Number(q.offset) || undefined, + }); if (!out.ok) return reply.code(400).send({ error: out.error }); return out.body; }); @@ -116,18 +131,35 @@ export function registerFindings(app: FastifyInstance, announce: Announce) { if (!requireToken(req, reply)) return; const id = idOf((req.params as { id: string }).id); if (id === null) return reply.code(404).send({ error: "no such finding" }); - const b = (req.body ?? {}) as { verdict?: unknown; note?: unknown; duplicate_of?: unknown }; - if (!("verdict" in b)) return reply.code(400).send({ error: "verdict is required (or null to reopen)" }); - if (b.note !== undefined && b.note !== null && typeof b.note !== "string") - return reply.code(400).send({ error: "note must be a string" }); - const dup = b.duplicate_of === undefined || b.duplicate_of === null ? null : Number(b.duplicate_of); - if (dup !== null && !(Number.isInteger(dup) && dup > 0)) - return reply.code(400).send({ error: "duplicate_of must be a finding id" }); - - const note = typeof b.note === "string" && b.note.trim() ? b.note.trim() : null; - const out = setVerdict(id, (b.verdict ?? null) as Verdict | null, note, dup); + const body = parseVerdictBody(req.body); + if (!body.ok) return reply.code(400).send({ error: body.error }); + const ctx = authOf(req); + const out = setVerdict(id, body.verdict, body.note, body.duplicateOf, { who: { actor: ctx.actor, source: ctx.source ?? "dash" } }); if (!out.ok) return reply.code(out.status).send({ error: out.error }); announce({ type: "finding", id, verdict: out.finding.verdict }); - return { ok: true, finding: out.finding }; + return { ok: true, finding: out.finding, regression: out.regression, change: out.change }; + }); + + // The executor's side of regression tests. Literal before parameterised, + // as above. + app.get("/findings/regressions", async (req, reply) => { + const q = req.query as Record; + const out = listRegressions({ app: q.app || undefined, state: q.state || undefined, platform: q.platform || undefined }); + if (!out.ok) return reply.code(400).send({ error: out.error }); + return { regressions: out.items }; + }); + + app.post("/findings/:id/regression", async (req, reply) => { + const id = idOf((req.params as { id: string }).id); + if (id === null) return reply.code(404).send({ error: "no such finding" }); + const body = parseRegressionRun(req.body); + if (!body.ok) return reply.code(400).send({ error: body.error }); + const out = recordRegressionRun(id, body.run); + if (!out.ok) return reply.code(out.status).send({ error: out.error }); + if (out.reopened) { + app.log.warn({ finding: id, job: body.run.job_id, build: body.run.build }, "regression reproduced; finding reopened"); + announce({ type: "finding", id, verdict: null, regressed: true }); + } + return { ok: true, reopened: out.reopened, regression: { state: out.regression.state, runs: out.regression.runs, last_outcome: out.regression.last_outcome } }; }); } diff --git a/collector/src/api/guard.ts b/collector/src/api/guard.ts index 20b621c..b0b54b7 100644 --- a/collector/src/api/guard.ts +++ b/collector/src/api/guard.ts @@ -17,6 +17,12 @@ const TOKEN = process.env.FLEET_DASH_TOKEN; export const guardEnabled = () => AUTH === "tokens" || !!TOKEN; +/** The same rule as requireToken, without answering: for /api/v1, which answers in its own envelope. */ +export function dashTokenOk(req: FastifyRequest): boolean { + if (AUTH === "tokens" || !TOKEN) return true; + return req.headers["x-fleet-token"] === TOKEN; +} + /** Returns true when the request may proceed; sends 401 and returns false when * it may not, so callers can `if (!requireToken(req, reply)) return;`. */ export function requireToken(req: FastifyRequest, reply: FastifyReply): boolean { diff --git a/collector/src/api/v1/deprecations.ts b/collector/src/api/v1/deprecations.ts index 6b5feba..30755e2 100644 --- a/collector/src/api/v1/deprecations.ts +++ b/collector/src/api/v1/deprecations.ts @@ -26,6 +26,11 @@ export const SUCCESSORS: [method: string, pattern: RegExp, successor: (m: RegExp ["GET", /^\/api\/results$/, () => "/api/v1/results"], ["GET", /^\/devices$/, () => "/api/v1/devices"], ["GET", /^\/schedules$/, () => "/api/v1/schedules"], + // The list, one finding and the verdict. /api/findings/precision, /digest + // and /issues have no successor, and the id pattern leaves them alone. + ["GET", /^\/api\/findings$/, () => "/api/v1/findings"], + ["GET", /^\/api\/findings\/(\d+)$/, (m) => `/api/v1/findings/${m[1]}`], + ["POST", /^\/api\/findings\/(\d+)\/verdict$/, (m) => `/api/v1/findings/${m[1]}/verdict`], ]; /** Must run before any route is declared, like every root hook. */ diff --git a/collector/src/api/v1/openapi.ts b/collector/src/api/v1/openapi.ts index 3798863..cede04d 100644 --- a/collector/src/api/v1/openapi.ts +++ b/collector/src/api/v1/openapi.ts @@ -104,6 +104,7 @@ export function openapiDocument() { { name: "jobs", description: "Queue, validate, wait for, cancel and retry work." }, { name: "results", description: "What ran and how it went." }, { name: "artifacts", description: "Screenshots, logs, reports, builds." }, + { name: "findings", description: "What night QA found, its verdicts, and the regression tests fixed findings became." }, { name: "fleet", description: "Schedules, alerts, audit." }, { name: "protocol", description: "What runners speak. Not for composing work." }, ], diff --git a/collector/src/api/v1/routes.ts b/collector/src/api/v1/routes.ts index 53defec..919ec62 100644 --- a/collector/src/api/v1/routes.ts +++ b/collector/src/api/v1/routes.ts @@ -23,7 +23,7 @@ import { createReadStream, existsSync, statSync } from "node:fs"; import { randomBytes } from "node:crypto"; import path from "node:path"; import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify"; -import { onAnnounce } from "../../announce.js"; +import { announce, onAnnounce } from "../../announce.js"; import { authOf, forward, type Scope } from "../../auth.js"; import { ARTIFACT_DIR, AUTH, AUTH_PROTOCOL } from "../../config.js"; import { APP_VERSION } from "../../version.js"; @@ -33,10 +33,12 @@ import { CATALOG, catalogEntry, type CatalogEntry } from "../../workloads/catalo import { isTextual } from "../../media-type.js"; import { AGE, capabilityMatches, deviceCapabilities, inClause, iso, isExpired, ONLINE_S, paging, parse, sha256Refs } from "../shared.js"; import { needsAdmin, notOwned } from "../policy.js"; +import { dashTokenOk } from "../guard.js"; import { schedulesView } from "../system.js"; import { SCHEMAS } from "./schemas.js"; import { isSkipError } from "../../skip.js"; import { errorsOf, paramsValidator, validateSpec, type SchemaError } from "./spec-check.js"; +import { findingDetail, listFindings, parseVerdictBody, setVerdict } from "../../findings.js"; // --- the table's types ------------------------------------------------------------ @@ -71,7 +73,7 @@ export type Route = { method: "GET" | "POST"; /** OpenAPI-style path: `/api/v1/jobs/{id}`. */ path: string; - tag: "meta" | "devices" | "workloads" | "jobs" | "results" | "artifacts" | "fleet" | "protocol"; + tag: "meta" | "devices" | "workloads" | "jobs" | "results" | "artifacts" | "findings" | "fleet" | "protocol"; summary: string; description?: string; scope: "public" | "protocol" | Scope; @@ -147,6 +149,12 @@ const PAGE: Param[] = [ { name: "per_page", in: "query", type: "integer", description: "Items per page.", minimum: 1, maximum: 500 }, ]; const JOB_ID: Param = { name: "id", in: "path", type: "string", required: true, description: "A job id, or a fan-out parent's id." }; +const FINDING_ID: Param = { name: "id", in: "path", type: "integer", required: true, description: "The finding's id." }; +/** A finding id from the path, or a 404 for anything that cannot be one. */ +const findingId = (raw: string) => { + if (!/^\d+$/.test(raw) || Number(raw) < 1) throw new ApiError(404, "not_found", `finding ${raw} not found`, "finding ids are positive integers"); + return Number(raw); +}; const SHA: Param = { name: "sha256", in: "path", type: "string", required: true, description: "The artifact's sha256 (64 hex)." }; const E400: Response = { status: 400, description: "The request was malformed or the spec is invalid.", schema: "Error" }; @@ -471,18 +479,19 @@ function listResults(q: Query) { type ArtifactRow = { sha256: string; name: string | null; size: number; media_type: string | null; app: string | null; build: string | null; - platform: string | null; created_at: string; published_at: string | null; pinned: number | null; + platform: string | null; created_at: string; published_at: string | null; pinned: number | null; channel: string | null; }; function artifactMeta(sha: string) { if (!/^[a-f0-9]{64}$/.test(sha)) throw new ApiError(400, "bad_request", "sha256 must be 64 lowercase hex characters"); const a = db - .prepare("SELECT sha256, name, size, media_type, app, build, platform, created_at, published_at, pinned FROM artifacts WHERE sha256 = ?") + .prepare("SELECT sha256, name, size, media_type, app, build, platform, created_at, published_at, pinned, channel FROM artifacts WHERE sha256 = ?") .get(sha) as ArtifactRow | undefined; if (!a) throw notFound("artifact"); return { sha256: a.sha256, name: a.name, size: a.size, media_type: a.media_type, app: a.app, build: a.build, platform: a.platform, created_at: iso(a.created_at), published_at: iso(a.published_at), pinned: !!a.pinned, + channel: a.channel === "pr" ? ("pr" as const) : ("main" as const), in_store: existsSync(path.join(ARTIFACT_DIR, a.sha256)), url: `/api/v1/artifacts/${a.sha256}/content`, }; } @@ -643,8 +652,19 @@ export const ROUTES: Route[] = [ body: { schema: "JobSpecInput", description: "The job spec.", required: true }, responses: [ { status: 201, description: "Queued (or waiting on dependencies).", schema: "JobCreated" }, - E400, E401, E403, E409, - { status: 422, description: "Valid, but nothing can run it (no capable agent, no matching device, no build).", schema: "Error" }, + { + status: 400, + description: + "Malformed or refused before queueing: invalid_spec, no_matching_devices (no registered device matches the targets), " + + "no_build (no artifact for app `latest`), invalid_match, invalid_dependency, secret_in_spec, bad_request.", + schema: "Error", + }, + E401, E403, E409, + { + status: 422, + description: "Valid, but cannot be queued: no_capable_agent (nothing declares the workload), unresolved_reference (a template reference that does not resolve).", + schema: "Error", + }, ], handler: async ({ req, reply }) => { const { spec } = prepareSpec(req); @@ -845,6 +865,72 @@ export const ROUTES: Route[] = [ }, }, + // findings + { + operationId: "listFindings", method: "GET", path: "/api/v1/findings", tag: "findings", scope: "read", readOnly: true, idempotent: true, + summary: "What night QA (the explore workload) found, newest sighting first.", + description: + "Each finding was replayed on a clean install before it was filed; `replay_label` says how often it reproduced. " + + "verdict agent_mistake means the model misread the app, not a bug in it. since=24h is the morning brief.", + params: [ + { name: "app", in: "query", type: "string", description: "Only this app (the explore app key)." }, + { name: "state", in: "query", type: "string", enum: ["open", "triaged", "all"], description: "open: no verdict yet (or reopened by a regression). Default all." }, + { name: "verdict", in: "query", type: "string", description: "Comma-separated: real, duplicate, not_a_bug, agent_mistake, fixed." }, + { name: "check", in: "query", type: "string", description: "crash, anr, frozen, blank, left_app, a11y, visual, goal, dead_control." }, + { name: "since", in: "query", type: "string", description: "Last seen at or after: ISO 8601, or a span back from now such as 90m, 24h, 7d." }, + ...PAGE, + ], + responses: [{ status: 200, description: "Findings, and the facets a filter menu is built from.", schema: "FindingList" }, E400], + handler: async ({ q }) => { + const { page, per_page, offset } = paging(q as Record, 50, 500); + const out = listFindings({ + app: q.app as string | undefined, status: q.state as string | undefined, + verdict: q.verdict as string | undefined, check: q.check as string | undefined, since: q.since as string | undefined, + limit: per_page, offset, + }); + if (!out.ok) throw new ApiError(400, "bad_request", out.error); + const b = out.body; + return { ...list(b.findings, page, per_page, b.total), facets: { apps: b.apps, checks: b.checks, counts: b.counts } }; + }, + }, + { + operationId: "getFinding", method: "GET", path: "/api/v1/findings/{id}", tag: "findings", scope: "read", readOnly: true, idempotent: true, + summary: "One finding: the steps in words, its evidence as links, the replay flow (inline when small), its regression test and verdict history.", + params: [FINDING_ID], + responses: [{ status: 200, description: "The finding.", schema: "FindingDetail" }, E404], + handler: async ({ p }) => findingDetail(findingId(p.id)) ?? (() => { throw notFound(`finding ${p.id}`, "GET /api/v1/findings lists them"); })(), + }, + { + operationId: "setFindingVerdict", method: "POST", path: "/api/v1/findings/{id}/verdict", tag: "findings", scope: "run", idempotent: true, + summary: "Judge a finding: real, duplicate, not_a_bug, agent_mistake, or fixed (only after real). null reopens.", + description: + "real keeps the finding's replay flow as a candidate regression test; fixed makes it active, so the app's nightly " + + "ui-test (with params.regressions) runs it. duplicate, not_a_bug and agent_mistake retire it. Mark fixed only once a fix " + + "has merged. Each verdict counts towards the check's precision, which decides what night QA runs tomorrow.", + params: [FINDING_ID], + body: { schema: "FindingVerdictInput", description: "The verdict and an optional note.", required: true }, + responses: [ + { status: 200, description: "Recorded.", schema: "FindingVerdictResult" }, + E400, E401, E403, E404, + { status: 409, description: "fixed on a finding that is not real (conflict).", schema: "Error" }, + ], + handler: async ({ req, p }) => { + const id = findingId(p.id); + // FLEET_DASH_TOKEN's speed bump, as on the dashboard's older route and on + // cancel (which forwards to a guarded route): a verdict decides what the + // nightly runs. + if (!dashTokenOk(req)) + throw new ApiError(401, "unauthorized", "X-Fleet-Token required (FLEET_DASH_TOKEN is set on the collector)", "send the dashboard token as X-Fleet-Token"); + const body = parseVerdictBody(req.body); + if (!body.ok) throw new ApiError(400, "bad_request", body.error); + const ctx = authOf(req); + const out = setVerdict(id, body.verdict, body.note, body.duplicateOf, { who: { actor: ctx.actor, source: ctx.source ?? "api" } }); + if (!out.ok) throw new ApiError(out.status, out.status === 404 ? "not_found" : out.status === 409 ? "conflict" : "bad_request", out.error, out.hint); + announce({ type: "finding", id, verdict: out.finding.verdict }); + return { finding: findingDetail(id), change: out.change }; + }, + }, + // fleet { operationId: "listSchedules", method: "GET", path: "/api/v1/schedules", tag: "fleet", scope: "read", readOnly: true, idempotent: true, @@ -913,9 +999,27 @@ export const ROUTES: Route[] = [ { operationId: "uploadArtifact", method: "POST", path: "/artifacts", tag: "protocol", scope: "protocol", documentOnly: true, summary: "Upload bytes (application/octet-stream). Answers with the sha256 and the detected media type.", - description: "Headers: x-artifact-name, x-artifact-media-type (optional), x-artifact-app, x-artifact-build.", + description: + "Headers: x-artifact-name, x-artifact-media-type (optional), x-artifact-app, x-artifact-build, " + + "x-artifact-channel (optional: `pr` for a pull request's build, which `\"sha256\": \"latest\"` never resolves to; absent means a merged build).", responses: [{ status: 201, description: "Stored.", schema: null }], }, + { + operationId: "postFinding", method: "POST", path: "/findings", tag: "protocol", scope: "protocol", documentOnly: true, + summary: "The explore workload filing a finding (an ExploreFinding). Upserts on the fingerprint.", + responses: [{ status: 201, description: "New.", schema: null }, { status: 200, description: "Seen again.", schema: null }], + }, + { + operationId: "listRegressionFlows", method: "GET", path: "/findings/regressions", tag: "protocol", scope: "protocol", documentOnly: true, + summary: "A nightly ui-test pulling an app's regression flows (?app= &state=active &platform=).", + responses: [{ status: 200, description: "{regressions: [...]}, each with its flow text.", schema: null }], + }, + { + operationId: "postRegressionRun", method: "POST", path: "/findings/{id}/regression", tag: "protocol", scope: "protocol", documentOnly: true, + summary: "An executor reporting one regression run: {job_id, device_id, build, outcome: passed|reproduced|failed|skipped, detail}. reproduced reopens the finding.", + params: [{ name: "id", in: "path", type: "integer", required: true, description: "The finding's id." }], + responses: [{ status: 200, description: "Recorded.", schema: null }], + }, { operationId: "downloadArtifact", method: "GET", path: "/artifacts/{sha256}", tag: "protocol", scope: "protocol", documentOnly: true, summary: "The bytes as application/octet-stream, with Range support. Runners fetch models and builds here.", diff --git a/collector/src/api/v1/schemas.ts b/collector/src/api/v1/schemas.ts index 8b5598e..07076f1 100644 --- a/collector/src/api/v1/schemas.ts +++ b/collector/src/api/v1/schemas.ts @@ -313,9 +313,14 @@ export const SCHEMAS: Record = { Artifact: object( { sha256: str, name: nstr, size: nint, media_type: nstr, app: nstr, build: nstr, platform: nstr, - created_at: nstr, published_at: nstr, pinned: bool, in_store: bool, url: str, + created_at: nstr, published_at: nstr, pinned: bool, + channel: { + enum: ["main", "pr"], + description: "pr: uploaded from a pull request (x-artifact-channel: pr); `\"sha256\": \"latest\"` never resolves to it. main: everything else.", + }, + in_store: bool, url: str, }, - ["sha256", "name", "size", "media_type", "in_store", "url"], + ["sha256", "name", "size", "media_type", "channel", "in_store", "url"], ), CancelResult: object( @@ -329,6 +334,144 @@ export const SCHEMAS: Record = { RetryResult: object({ job_id: str, retry_of: str }), + Finding: object( + { + id: int, fingerprint: str, app: { ...str, description: "The explore app key the finding was filed under." }, + build: { ...str, description: "The build it was first seen on." }, platform: str, device_id: str, + job_id: { ...str, description: "The job that first found it." }, last_job_id: nstr, mission_id: str, + check: { type: "string", description: "crash, anr, frozen, blank, left_app, a11y, visual, goal, dead_control." }, + severity: { enum: ["high", "medium", "low"] }, title: str, detail: str, screen: str, screen_name: str, + steps: { ...strs, description: "What to do to see it, in words, one step per entry." }, + replay: { + oneOf: [ + { type: "null" }, + object({ kind: { enum: ["maestro", "tvloop", "none"] }, sha256: nstr, attempts: int, reproduced: int }), + ], + description: "The replay on a clean install: how many of `attempts` reproduced it. null only for a crash, whose crash log is the evidence.", + }, + replay_label: { ...str, description: "reproduced 2/2, flaky 1/2, crash log." }, + artifacts: { + ...object({ shot: str, sheet: str, trajectory: str, log: str, replay: str }, []), + description: "Artifact sha256s by role: shot, sheet, trajectory, log, replay. Absent roles were not posted.", + }, + first_seen: nstr, last_seen: nstr, seen_count: int, builds_seen: strs, + verdict: { + enum: ["real", "duplicate", "not_a_bug", "agent_mistake", "fixed", null], + description: "null: nobody has judged it, or a regression test reopened it. agent_mistake is NOT a bug in the app.", + }, + verdict_note: nstr, verdict_at: nstr, duplicate_of: nint, + issue: { + oneOf: [ + { type: "null" }, + object({ + state: { enum: ["dry_run", "pending", "filed", "failed", "capped", "no_repo"] }, + dry_run: { ...bool, description: "True means nothing was sent to GitHub." }, + repo: nstr, title: str, body: str, labels: strs, url: nstr, number: nint, detail: str, at: str, + }), + ], + description: "The GitHub issue it would file, or did. Fleet Runner only ever creates issues; it never reads them back.", + }, + status: { enum: ["open", "triaged"] }, + regression: { + oneOf: [ + { type: "null" }, + object({ state: { enum: ["candidate", "active", "retired"] }, kind: { enum: ["maestro", "tvloop"] }, last_outcome: nstr }), + ], + description: "The regression test it became: candidate once real, active (in the nightly) once fixed.", + }, + }, + [ + "id", "fingerprint", "app", "build", "platform", "device_id", "job_id", "mission_id", "check", "severity", "title", "detail", + "screen_name", "steps", "replay", "replay_label", "artifacts", "seen_count", "builds_seen", "verdict", "status", + "first_seen", "last_seen", "regression", + ], + ), + + FindingList: listWith("Finding", { + facets: { + ...object({ + apps: strs, checks: strs, + counts: object({ open: int, triaged: int, total: int }), + }), + description: "apps and checks over every finding, so a filter never empties its own menu; counts over the app, check and since filters, before state and verdict.", + }, + }), + + FindingDetail: { + allOf: [ + ref("Finding"), + object( + { + duplicates: arr(object({ id: int, title: str, seen_count: int })), + duplicate_of_title: nstr, + counts: object({ seen: int, builds: int, replay_attempts: nint, replay_reproduced: nint, regression_runs: int }), + evidence: arr(object({ + role: { enum: ["shot", "sheet", "trajectory", "log", "replay"] }, label: str, sha256: str, + url: { ...str, description: "The bytes, relative to the collector." }, in_store: bool, + })), + replay_flow: { + oneOf: [ + { type: "null" }, + object({ + kind: str, sha256: str, url: str, in_store: bool, + text: { ...nstr, description: "The flow itself when it is at most 16 KiB; otherwise fetch url." }, + truncated: { ...bool, description: "True when the flow was too big to inline." }, + }), + ], + description: "The Maestro (or tvloop) flow that replays the finding's steps.", + }, + regression: { + oneOf: [ + { type: "null" }, + object( + { + finding_id: int, app: str, platform: str, check: str, + kind: { enum: ["maestro", "tvloop"] }, state: { enum: ["candidate", "active", "retired"] }, + asserts: { enum: ["no_crash", "replays"], description: "no_crash for crash and anr findings; replays (without error) for every other check." }, + flow: { ...str, description: "The flow text the nightly runs, copied when the finding was marked real." }, + flow_sha256: nstr, created_at: nstr, activated_at: nstr, retired_at: nstr, + runs: int, last_run_at: nstr, last_job_id: nstr, + last_outcome: { enum: ["passed", "reproduced", "failed", "skipped", null] }, last_detail: nstr, + }, + ["finding_id", "state", "kind", "asserts", "flow", "runs"], + ), + ], + }, + history: arr(object( + { + at: nstr, event: { enum: ["verdict", "regressed"] }, + verdict: { enum: ["real", "duplicate", "not_a_bug", "agent_mistake", "fixed", null] }, + note: nstr, actor: nstr, source: nstr, + }, + ["at", "event", "verdict", "note"], + )), + links: object({ self: str, verdict: str, dashboard: str }), + }, + ["duplicates", "duplicate_of_title", "counts", "evidence", "replay_flow", "regression", "history", "links"], + ), + ], + }, + + FindingVerdictInput: object( + { + verdict: { + enum: ["real", "duplicate", "not_a_bug", "agent_mistake", "fixed", null], + description: "fixed only after real, once the fix has merged. null reopens.", + }, + note: { type: ["string", "null"], maxLength: 2000 }, + duplicate_of: { ...nint, description: "Required with duplicate: the finding this one duplicates." }, + }, + ["verdict"], + { additionalProperties: false }, + ), + + FindingVerdictResult: object( + { + finding: ref("FindingDetail"), + change: { ...nstr, description: "What the verdict did to the finding's regression test, in one sentence, or null." }, + }, + ), + AuditEntry: object( { id: int, at: nstr, actor: nstr, verified: bool, token_id: nstr, source: nstr, diff --git a/collector/src/auth.ts b/collector/src/auth.ts index efead8e..01d2aad 100644 --- a/collector/src/auth.ts +++ b/collector/src/auth.ts @@ -239,6 +239,9 @@ const PROTOCOL_ROUTES: [string, RegExp][] = [ ["GET", /^\/events\/[^/]+\/poll$/], ["POST", /^\/api\/jobs\/[^/]+\/mirror$/], ["POST", /^\/findings$/], + // A nightly ui-test pulling the app's regression flows, and reporting each run. + ["GET", /^\/findings\/regressions$/], + ["POST", /^\/findings\/\d+\/regression$/], ]; const RUN_ROUTES: [string, RegExp][] = [ @@ -249,6 +252,9 @@ const RUN_ROUTES: [string, RegExp][] = [ ["POST", /^\/api\/v1\/jobs$/], ["POST", /^\/api\/v1\/jobs\/validate$/], ["POST", /^\/api\/v1\/jobs\/[^/]+\/(cancel|retry)$/], + // A verdict decides what tomorrow's nightly runs, which is the run scope's + // business; the dashboard's older route stays where it was. + ["POST", /^\/api\/v1\/findings\/[^/]+\/verdict$/], ]; /** What a request needs. Exported so the rule table can be tested and documented. */ @@ -356,7 +362,7 @@ function deny(req: FastifyRequest, reply: FastifyReply, status: 401 | 403, code: } /** Routes whose traffic is the fleet breathing, not a person deciding. Never audited. */ -const QUIET = /^\/(results|devices\/register|devices\/[^/]+\/next-job|executor\/next-job|locks\/|events\/|artifacts$|findings$|api\/jobs\/[^/]+\/mirror$|api\/session$)/; +const QUIET = /^\/(results|devices\/register|devices\/[^/]+\/next-job|executor\/next-job|locks\/|events\/|artifacts$|findings$|findings\/\d+\/regression$|api\/jobs\/[^/]+\/mirror$|api\/session$)/; export function registerAuth(app: FastifyInstance): void { app.addHook("onRequest", async (req, reply) => { diff --git a/collector/src/client/api.gen.ts b/collector/src/client/api.gen.ts index f83fc40..f6d883c 100644 --- a/collector/src/client/api.gen.ts +++ b/collector/src/client/api.gen.ts @@ -33,6 +33,8 @@ export type Artifact = { created_at?: string | null; published_at?: string | null; pinned?: boolean; + /** pr: uploaded from a pull request (x-artifact-channel: pr); `"sha256": "latest"` never resolves to it. main: everything else. */ + channel: "main" | "pr"; in_store: boolean; url: string; }; @@ -201,6 +203,178 @@ export type Error = { }; }; +export type Finding = { + id: number; + fingerprint: string; + /** The explore app key the finding was filed under. */ + app: string; + /** The build it was first seen on. */ + build: string; + platform: string; + device_id: string; + /** The job that first found it. */ + job_id: string; + last_job_id?: string | null; + mission_id: string; + /** crash, anr, frozen, blank, left_app, a11y, visual, goal, dead_control. */ + check: string; + severity: "high" | "medium" | "low"; + title: string; + detail: string; + screen?: string; + screen_name: string; + /** What to do to see it, in words, one step per entry. */ + steps: string[]; + /** The replay on a clean install: how many of `attempts` reproduced it. null only for a crash, whose crash log is the evidence. */ + replay: (null | { + kind: "maestro" | "tvloop" | "none"; + sha256: string | null; + attempts: number; + reproduced: number; + }); + /** reproduced 2/2, flaky 1/2, crash log. */ + replay_label: string; + /** Artifact sha256s by role: shot, sheet, trajectory, log, replay. Absent roles were not posted. */ + artifacts: { + shot?: string; + sheet?: string; + trajectory?: string; + log?: string; + replay?: string; + }; + first_seen: string | null; + last_seen: string | null; + seen_count: number; + builds_seen: string[]; + /** null: nobody has judged it, or a regression test reopened it. agent_mistake is NOT a bug in the app. */ + verdict: "real" | "duplicate" | "not_a_bug" | "agent_mistake" | "fixed" | null; + verdict_note?: string | null; + verdict_at?: string | null; + duplicate_of?: number | null; + /** The GitHub issue it would file, or did. Fleet Runner only ever creates issues; it never reads them back. */ + issue?: (null | { + state: "dry_run" | "pending" | "filed" | "failed" | "capped" | "no_repo"; + /** True means nothing was sent to GitHub. */ + dry_run: boolean; + repo: string | null; + title: string; + body: string; + labels: string[]; + url: string | null; + number: number | null; + detail: string; + at: string; + }); + status: "open" | "triaged"; + /** The regression test it became: candidate once real, active (in the nightly) once fixed. */ + regression: (null | { + state: "candidate" | "active" | "retired"; + kind: "maestro" | "tvloop"; + last_outcome: string | null; + }); +}; + +export type FindingDetail = Finding & { + duplicates: { + id: number; + title: string; + seen_count: number; + }[]; + duplicate_of_title: string | null; + counts: { + seen: number; + builds: number; + replay_attempts: number | null; + replay_reproduced: number | null; + regression_runs: number; + }; + evidence: { + role: "shot" | "sheet" | "trajectory" | "log" | "replay"; + label: string; + sha256: string; + /** The bytes, relative to the collector. */ + url: string; + in_store: boolean; + }[]; + /** The Maestro (or tvloop) flow that replays the finding's steps. */ + replay_flow: (null | { + kind: string; + sha256: string; + url: string; + in_store: boolean; + /** The flow itself when it is at most 16 KiB; otherwise fetch url. */ + text: string | null; + /** True when the flow was too big to inline. */ + truncated: boolean; + }); + regression: (null | { + finding_id: number; + app?: string; + platform?: string; + check?: string; + kind: "maestro" | "tvloop"; + state: "candidate" | "active" | "retired"; + /** no_crash for crash and anr findings; replays (without error) for every other check. */ + asserts: "no_crash" | "replays"; + /** The flow text the nightly runs, copied when the finding was marked real. */ + flow: string; + flow_sha256?: string | null; + created_at?: string | null; + activated_at?: string | null; + retired_at?: string | null; + runs: number; + last_run_at?: string | null; + last_job_id?: string | null; + last_outcome?: "passed" | "reproduced" | "failed" | "skipped" | null; + last_detail?: string | null; + }); + history: { + at: string | null; + event: "verdict" | "regressed"; + verdict: "real" | "duplicate" | "not_a_bug" | "agent_mistake" | "fixed" | null; + note: string | null; + actor?: string | null; + source?: string | null; + }[]; + links: { + self: string; + verdict: string; + dashboard: string; + }; +}; + +export type FindingList = { + items: Finding[]; + page: number; + per_page: number; + total: number; + pages: number; + /** apps and checks over every finding, so a filter never empties its own menu; counts over the app, check and since filters, before state and verdict. */ + facets: { + apps: string[]; + checks: string[]; + counts: { + open: number; + triaged: number; + total: number; + }; + }; +}; + +export type FindingVerdictInput = { + /** fixed only after real, once the fix has merged. null reopens. */ + verdict: "real" | "duplicate" | "not_a_bug" | "agent_mistake" | "fixed" | null; + note?: string | null; + /** Required with duplicate: the finding this one duplicates. */ + duplicate_of?: number | null; +}; + +export type FindingVerdictResult = { + finding: FindingDetail; + /** What the verdict did to the finding's regression test, in one sentence, or null. */ + change: string | null; +}; + export type FlakyCell = { /** ` ` */ build: string; @@ -380,7 +554,7 @@ export type JobSpecInput = { kind?: "maestro" | "xcuitest" | "playwright"; flows?: string; }; - /** Workload-specific. Deliberately open, but the fields the shipped workloads read are listed here so they are discoverable: benchmark/thermal prompt_tokens, gen_tokens, warmup_iters, measure_iters, duration_s; batch/vision/speech/embed input_sha256, max_items, max_tokens; cold-start app_id, launches, states; locale-shots locales; app-soak flow, duration_s, interval_s; a11y-audit steps, variants, min_target_pt, font_scale, tree_source; build repo, ref, kind, task, scheme; vantage urls, repeats; any host job network and network_to; a preempted job resume_from. */ + /** Workload-specific. Deliberately open, but the fields the shipped workloads read are listed here so they are discoverable: benchmark/thermal prompt_tokens, gen_tokens, warmup_iters, measure_iters, duration_s; batch/vision/speech/embed input_sha256, max_items, max_tokens; cold-start app_id, launches, states; locale-shots locales; app-soak flow, duration_s, interval_s; a11y-audit steps, variants, min_target_pt, font_scale, tree_source; build repo, ref, kind, task, scheme; vantage urls, repeats; ui-test regressions (true, or the app its findings are filed under: run its fixed findings' replay flows too); any host job network and network_to; a preempted job resume_from. */ params?: Record; /** Job ids this one waits for. The job is enqueued as 'waiting' and becomes 'queued' when the last of them finishes; if any of them fails or is cancelled, this one fails with last_error naming it. While waiting, a spec may reference a dependency's output: "${jobs..artifact}" resolves to the first sha256 in that job's final row, and "${jobs..metrics.}" to one of its metrics. A whole-string reference keeps the value's type, so a metric stays a number. Referencing a job not listed here is refused at enqueue. */ depends_on?: string[]; @@ -816,6 +990,42 @@ export class FleetClient extends Transport { return this.request("GET", `/api/v1/artifacts/${enc(sha256)}/content`, { ...opts, query, raw: true }); } + /** + * What night QA (the explore workload) found, newest sighting first. + * + * Each finding was replayed on a clean install before it was filed; `replay_label` says how often it + * reproduced. verdict agent_mistake means the model misread the app, not a bug in it. since=24h is the + * morning brief. + * + * `GET /api/v1/findings` · scope: read + */ + listFindings(query?: { app?: string; state?: "open" | "triaged" | "all"; verdict?: string; check?: string; since?: string; page?: number; per_page?: number }, opts?: CallOptions): Promise { + return this.request("GET", `/api/v1/findings`, { ...opts, query }); + } + + /** + * One finding: the steps in words, its evidence as links, the replay flow (inline when small), its regression test and verdict history. + * + * `GET /api/v1/findings/{id}` · scope: read + */ + getFinding(id: string, opts?: CallOptions): Promise { + return this.request("GET", `/api/v1/findings/${enc(id)}`, { ...opts }); + } + + /** + * Judge a finding: real, duplicate, not_a_bug, agent_mistake, or fixed (only after real). null reopens. + * + * real keeps the finding's replay flow as a candidate regression test; fixed makes it active, so the + * app's nightly ui-test (with params.regressions) runs it. duplicate, not_a_bug and agent_mistake + * retire it. Mark fixed only once a fix has merged. Each verdict counts towards the check's precision, + * which decides what night QA runs tomorrow. + * + * `POST /api/v1/findings/{id}/verdict` · scope: run + */ + setFindingVerdict(id: string, body: FindingVerdictInput, opts?: CallOptions): Promise { + return this.request("POST", `/api/v1/findings/${enc(id)}/verdict`, { ...opts, body }); + } + /** * Cron schedules, with next run and whether one was missed. * diff --git a/collector/src/db.ts b/collector/src/db.ts index b1fc04d..d7cecc5 100644 --- a/collector/src/db.ts +++ b/collector/src/db.ts @@ -676,7 +676,7 @@ CREATE TABLE IF NOT EXISTS findings ( last_seen TEXT NOT NULL DEFAULT (datetime('now')), seen_count INTEGER NOT NULL DEFAULT 1, builds_seen TEXT NOT NULL DEFAULT '[]', -- JSON array of distinct builds - verdict TEXT CHECK (verdict IS NULL OR verdict IN ('real','duplicate','not_a_bug','agent_mistake')), + verdict TEXT CHECK (verdict IS NULL OR verdict IN ('real','duplicate','not_a_bug','agent_mistake','fixed')), verdict_note TEXT, verdict_at TEXT, duplicate_of INTEGER, @@ -701,8 +701,84 @@ CREATE TABLE IF NOT EXISTS findings_digests ( posted INTEGER NOT NULL DEFAULT 0, detail TEXT ); + +-- Every verdict a finding has had, and every night a regression test brought +-- it back. The findings row holds only the current verdict; this is how a +-- reader learns that "open" means "regressed on build 1.4 (12)" rather than +-- "nobody has looked". +CREATE TABLE IF NOT EXISTS finding_verdicts ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + finding_id INTEGER NOT NULL, + at TEXT NOT NULL DEFAULT (datetime('now')), + event TEXT NOT NULL CHECK (event IN ('verdict','regressed')), + verdict TEXT, -- NULL: reopened + note TEXT, + actor TEXT, + source TEXT +); +CREATE INDEX IF NOT EXISTS idx_finding_verdicts ON finding_verdicts (finding_id, id); + +-- A finding's replay flow, kept as a regression test (src/findings.ts). +-- In the database rather than as files because the lab's flows directory is +-- outside git and outside any backup this repository controls: the flow text +-- is copied here when the finding is marked real, so it survives the artifact +-- store's GC as well. +CREATE TABLE IF NOT EXISTS regressions ( + finding_id INTEGER PRIMARY KEY, + app TEXT NOT NULL, + platform TEXT NOT NULL, + check_name TEXT NOT NULL, + kind TEXT NOT NULL CHECK (kind IN ('maestro','tvloop')), + flow TEXT NOT NULL, + flow_sha256 TEXT, + state TEXT NOT NULL CHECK (state IN ('candidate','active','retired')), + created_at TEXT NOT NULL DEFAULT (datetime('now')), + activated_at TEXT, + retired_at TEXT, + runs INTEGER NOT NULL DEFAULT 0, + last_run_at TEXT, + last_job_id TEXT, + last_outcome TEXT, + last_detail TEXT +); +CREATE INDEX IF NOT EXISTS idx_regressions_app ON regressions (app, state); `); +// 'fixed' as a verdict. The findings table's CHECK predates it, and SQLite +// cannot alter a constraint, so the table is rebuilt once, the same way the +// jobs table was for 'skipped'. Every existing verdict also gets one history +// row, so a finding judged before history existed does not read as never +// judged. +{ + const ddl = (db.prepare("SELECT sql FROM sqlite_master WHERE type='table' AND name='findings'").get() as + | { sql: string } + | undefined)?.sql; + if (ddl && !ddl.includes("'fixed'")) { + const widened = ddl + .replace(/^CREATE TABLE\s+("?)findings\1/i, "CREATE TABLE findings_migrating") + .replace(/verdict IN \(([^)]*)\)/i, (_m, list: string) => `verdict IN (${list.trim()},'fixed')`); + if (!widened.includes("'fixed'") || !widened.startsWith("CREATE TABLE findings_migrating")) + throw new Error("cannot widen the findings verdict constraint: the table's DDL is not the shape this migration knows"); + const columns = (db.prepare("PRAGMA table_info(findings)").all() as { name: string }[]).map((c) => `"${c.name}"`).join(", "); + withTransaction(db, () => { + db.exec(widened); + db.exec(`INSERT INTO findings_migrating (${columns}) SELECT ${columns} FROM findings`); + db.exec("DROP TABLE findings"); + db.exec("ALTER TABLE findings_migrating RENAME TO findings"); + db.exec(` + CREATE INDEX IF NOT EXISTS idx_findings_last_seen ON findings (last_seen DESC); + CREATE INDEX IF NOT EXISTS idx_findings_app_check ON findings (app, check_name); + CREATE INDEX IF NOT EXISTS idx_findings_issue ON findings (app, issue_state, issue_at); + `); + })(); + } + db.exec(` + INSERT INTO finding_verdicts (finding_id, at, event, verdict, note) + SELECT f.id, COALESCE(f.verdict_at, f.last_seen), 'verdict', f.verdict, f.verdict_note FROM findings f + WHERE f.verdict IS NOT NULL AND NOT EXISTS (SELECT 1 FROM finding_verdicts v WHERE v.finding_id = f.id) + `); +} + // Who asked for a job, and through which front door. After the rebuild for the // same reason depends_on is. `requested_by` is a verified identity only when // FLEET_AUTH=tokens; with auth off it is whatever the caller said, and the API diff --git a/collector/src/executor.ts b/collector/src/executor.ts index 10303e3..31c0537 100644 --- a/collector/src/executor.ts +++ b/collector/src/executor.ts @@ -55,6 +55,7 @@ import { FLOWS_DIR, MAESTRO, resolveFlow, runFlow } from "./workloads/flows.js"; import { discoverWorkloads, resolveRun, type LoadedWorkload } from "./workloads/registry.js"; import { bundledNames } from "./workloads/static.js"; import { prepareInstallable } from "./workloads/install/index.js"; +import { fetchRegressions, regressionApp, runRegressions, type RegressionFlow, type RegressionResult } from "./workloads/regressions.js"; import type { Target, WorkloadCtx } from "./workloads/types.js"; @@ -500,9 +501,13 @@ async function runUiTest(job: Job) { if (!suite) throw new Error("ui-test job needs a suite"); if (suite.kind === "xcuitest") return runXcuitest(job); if (suite.kind !== "maestro") throw new Error(`suite kind ${suite.kind} not supported yet`); - if (!suite.flows) throw new Error("maestro suite needs flows"); - const flows = path.resolve(FLOWS_DIR, suite.flows); - if (!existsSync(flows)) throw new Error(`flows not found: ${flows}`); + // params.regressions: the app's fixed findings, replayed after the suite's + // own flow on every device (src/workloads/regressions.ts). With it, a job + // may have no flow of its own and run only those. + const regressionsFor = regressionApp(job); + if (!suite.flows && !regressionsFor) throw new Error("maestro suite needs flows (or params.regressions to run only an app's regression flows)"); + const flows = suite.flows ? path.resolve(FLOWS_DIR, suite.flows) : null; + if (flows && !existsSync(flows)) throw new Error(`flows not found: ${flows}`); // Maestro takes `-e KEY=value` and a flow reads it as ${KEY}, which is how a // flow types into a login form. Same Keychain, same rule: the job names the @@ -524,14 +529,22 @@ async function runUiTest(job: Job) { // one flow that existed could never pass against the installed package. // A flow written as `appId: ${APP_ID}` takes the id from the job instead, and // the same flow then covers debug, smoke and release. - const appIdMatch = /^appId:\s*(\S+)/m.exec(readFileSync(flows, "utf8")); + const appIdMatch = flows ? /^appId:\s*(\S+)/m.exec(readFileSync(flows, "utf8")) : null; const declared = appIdMatch?.[1]; const parameterised = !declared || /\$\{|^\$/.test(declared); - if (parameterised && !job.suite?.app_id) { + if (flows && parameterised && !job.suite?.app_id) { throw new Error(`${flows} takes its appId from the job, so suite.app_id is required`); } const appId = job.suite?.app_id ?? declared; + // Pulled once, at run time: a finding marked fixed yesterday is in + // tonight's run without anyone editing this job's schedule. + let regressionFlows: RegressionFlow[] = []; + if (regressionsFor) { + regressionFlows = await fetchRegressions(regressionsFor); + log(`ui-test: ${regressionFlows.length} active regression flow(s) for ${regressionsFor}`); + } + // A job that names a build tests that build. Without this a ui-test ran its // flow against whatever was already on the phone -- so a CI run "pinned to // the exact APK just built" passed or failed on last week's install, and a @@ -590,6 +603,14 @@ async function runUiTest(job: Job) { log(`ui-test on ${serial}: skipped (${appId} not installed)`); continue; } + const mineRegs = regressionFlows.filter((f) => f.platform === target.platform); + if (!flows && mineRegs.length === 0) { + await postResult({ + job_id: job.job_id, device_id: serial, iter: 0, ok: true, + error: `skipped: no active regression flows for ${regressionsFor} on ${target.platform}`, + }); + continue; + } const outDir = mkdtempSync(path.join(os.tmpdir(), "fleet-junit-")); const report = path.join(outDir, "report.xml"); let failedToRun = false; @@ -603,7 +624,7 @@ async function runUiTest(job: Job) { // unshaped — an offline suite that quietly ran online is a pass that proves // nothing. let shapeError: string | undefined; - try { + if (flows) try { await withNetwork(job, target, async () => { try { await exec( @@ -628,7 +649,7 @@ async function runUiTest(job: Job) { } let passed = 0; - let failed = 1; + let failed = flows ? 1 : 0; let failure: string | null = null; let failingFlow: string | undefined; const artifacts: string[] = []; @@ -647,15 +668,40 @@ async function runUiTest(job: Job) { if (creds) writeFileSync(report, redact(xml, [creds.password])); artifacts.push(await uploadArtifact(report, `${job.job_id}-${serial}-junit.xml`)); } - if (failed > 0) allOk = false; - const error = shapeError + const suiteError = shapeError ? `params.network: ${shapeError}` : failedToRun ? `maestro produced no report: ${runError ?? "it never ran"}` : failure ?? undefined; + + // The app's regression flows, after its own. A reproduced crash reopens + // its finding on the collector and fails this row by name. + let regs: RegressionResult[] = []; + if (mineRegs.length) { + regs = await runRegressions({ + job, target, flows: mineRegs, appId, + env: creds ? { [creds.emailVar]: creds.account, [creds.passwordVar]: creds.password } : {}, + secrets: creds ? [creds.password] : [], + }); + passed += regs.filter((r) => r.outcome === "passed").length; + failed += regs.filter((r) => r.outcome === "reproduced" || r.outcome === "failed").length; + for (const r of regs) log(`ui-test on ${serial}: regression #${r.finding_id} ${r.outcome}: ${r.detail}`); + } + const badRegs = [...regs.filter((r) => r.outcome === "reproduced"), ...regs.filter((r) => r.outcome === "failed")]; + if (!failingFlow && badRegs.length) failingFlow = `regression finding #${badRegs[0].finding_id}`; + const regressionError = badRegs.length + ? badRegs.map((r) => `regression #${r.finding_id} ${r.outcome === "reproduced" ? "reproduced (finding reopened)" : "failed"}: ${r.detail}`).join("; ") + : undefined; + const error = [suiteError, regressionError].filter(Boolean).join("; ").slice(0, 1000) || undefined; + + if (failed > 0) allOk = false; await postResult({ job_id: job.job_id, device_id: serial, iter: 0, - ok: failed === 0, test: { passed, failed, artifacts, ...(failingFlow ? { failing_flow: failingFlow } : {}) }, + ok: failed === 0, + test: { + passed, failed, artifacts, ...(failingFlow ? { failing_flow: failingFlow } : {}), + ...(regs.length ? { regressions: regs } : {}), + }, ...(error ? { error } : {}), }); log(`ui-test on ${serial}: ${passed} passed / ${failed} failed${shapeError ? ` (network shaping: ${shapeError})` : ""}`); diff --git a/collector/src/findings.test.ts b/collector/src/findings.test.ts index 9955d8f..922787e 100644 --- a/collector/src/findings.test.ts +++ b/collector/src/findings.test.ts @@ -8,7 +8,8 @@ * loopback port -- nothing here ever reaches the real one), and the digest * timer's once-a-day rule, which depends on the wall clock. */ -import { mkdtempSync, rmSync } from "node:fs"; +import { mkdirSync, mkdtempSync, rmSync } from "node:fs"; +import { DatabaseSync } from "node:sqlite"; import { createServer, type IncomingMessage, type Server } from "node:http"; import { tmpdir } from "node:os"; import path from "node:path"; @@ -16,6 +17,14 @@ import { closeDb, db, openDb } from "./db.js"; import { ISSUE_CAP_PER_DAY, buildDigest, + findingDetail, + getRegression, + listFindings, + listRegressions, + parseRegressionRun, + parseSince, + parseVerdictBody, + recordRegressionRun, digestDue, digestTick, fileIssue, @@ -355,4 +364,132 @@ export async function runFindingsChecks(check: Check) { await down.close(); } }); + + // --- fixed, the verdict history, and a finding becoming a regression test --- + await inTempDb(() => { + const FLOW = "appId: com.example.plants\nname: \"Save does nothing\"\n---\n- launchApp:\n clearState: true\n- tapOn:\n point: \"50%,90%\"\n"; + const store = new Map([[SHA("a"), FLOW]]); + const read = (sha: string) => store.get(sha) ?? null; + const who = { actor: "taylor", source: "test" }; + + const crash = storeFinding(sample({ fingerprint: "r-crash", check: "crash" }), DRY).finding; + const early = setVerdict(crash.id, "fixed", "too soon", null, { who, readArtifact: read }); + check("regressions: fixed before real is refused with 409", !early.ok && early.status === 409 && /after real/.test(early.error), JSON.stringify(early)); + + const real = setVerdict(crash.id, "real", "seen it", null, { who, readArtifact: read }); + check("regressions: real keeps the replay flow as a candidate", + real.ok && real.regression?.state === "candidate" && real.regression.flow === FLOW && real.regression.kind === "maestro" + && real.regression.asserts === "no_crash" && /candidate/.test(real.change ?? ""), JSON.stringify(real)); + check("regressions: a candidate is not in the active list", listRegressions({ app: "plants" }).ok && + (listRegressions({ app: "plants" }) as { items: unknown[] }).items.length === 0); + const fixed = setVerdict(crash.id, "fixed", "PR #12", null, { who, readArtifact: read }); + check("regressions: fixed after real makes it active", + fixed.ok && fixed.finding.verdict === "fixed" && fixed.regression?.state === "active" && fixed.regression.activated_at !== null); + const active = listRegressions({ app: "plants", platform: "android" }); + check("regressions: the nightly's pull lists it with its flow", + active.ok && active.items.length === 1 && active.items[0].finding_id === crash.id && active.items[0].flow === FLOW); + check("regressions: listing refuses a state it does not know", !listRegressions({ state: "sideways" }).ok); + check("regressions: fixed counts as real in precision", precision({ app: "plants" }).classes.find((c) => c.check === "crash")?.real === 1); + + const job = { job_id: "nightly-ui-1", device_id: "pixel", build: "1.1 (5)", detail: "" }; + const passed = recordRegressionRun(crash.id, { ...job, outcome: "passed", detail: "no crash" }); + check("regressions: a pass is recorded and changes nothing else", + passed.ok && !passed.reopened && passed.finding.verdict === "fixed" && passed.regression.runs === 1 && passed.regression.last_outcome === "passed"); + const back = recordRegressionRun(crash.id, { ...job, job_id: "nightly-ui-2", outcome: "reproduced", detail: "the app crashed again during the flow: NPE" }); + check("regressions: reproduced reopens the finding, counts it and adds the build", + back.ok && back.reopened && back.finding.verdict === null && back.finding.status === "open" && back.finding.seen_count === crash.seen_count + 1 + && back.finding.builds_seen.includes("1.1 (5)") && back.regression.state === "active", JSON.stringify(back)); + const twice = recordRegressionRun(crash.id, { ...job, job_id: "nightly-ui-2", device_id: "galaxy", outcome: "reproduced", detail: "again" }); + check("regressions: a second device in the same job does not reopen it twice", twice.ok && !twice.reopened && twice.finding.seen_count === crash.seen_count + 1); + check("regressions: a run for a finding with no test is 404", (() => { + const r = recordRegressionRun(99999, { ...job, outcome: "passed" }); return !r.ok && r.status === 404; + })()); + + const detail = findingDetail(crash.id, read)!; + const events = detail.history.map((h) => `${h.event}:${h.verdict ?? "-"}`).join(","); + check("history: every verdict and the regression, in order", events === "verdict:real,verdict:fixed,regressed:-", events); + check("history: says which build it regressed on and who judged it", + /regressed on build 1\.1 \(5\)/.test(detail.history[2].note ?? "") && detail.history[0].actor === "taylor", JSON.stringify(detail.history)); + check("detail: the replay flow is inline, and the evidence links to /api/v1", + detail.replay_flow?.text === FLOW && detail.evidence.some((e) => e.role === "shot" && e.url === `/api/v1/artifacts/${SHA("b")}/content`), + JSON.stringify(detail.replay_flow)); + check("detail: counts", detail.counts.seen === crash.seen_count + 1 && detail.counts.regression_runs === 3, JSON.stringify(detail.counts)); + + const refixed = setVerdict(crash.id, "fixed", "PR #13", null, { readArtifact: read }); + check("regressions: a regressed finding may be marked fixed again directly", refixed.ok && refixed.regression?.state === "active"); + const retired = setVerdict(crash.id, "not_a_bug", null, null, { readArtifact: read }); + check("regressions: not_a_bug retires an active test", retired.ok && retired.regression?.state === "retired" && /retired/.test(retired.change ?? "")); + const again = setVerdict(crash.id, "real", null, null, { readArtifact: read }); + check("regressions: real brings a retired test back as a candidate", again.ok && again.regression?.state === "candidate"); + + const gone = storeFinding(sample({ fingerprint: "r-gone", replay: { kind: "maestro", sha256: SHA("9"), attempts: 2, reproduced: 2 }, artifacts: {} }), DRY).finding; + const noFlow = setVerdict(gone.id, "real", null, null, { readArtifact: read }); + check("regressions: real with no stored flow says why and keeps no test", + noFlow.ok && noFlow.regression === null && /not in the artifact store/.test(noFlow.change ?? ""), JSON.stringify(noFlow)); + const fixedNoFlow = setVerdict(gone.id, "fixed", null, null, { readArtifact: read }); + check("regressions: fixed with no flow is still recorded", fixedNoFlow.ok && fixedNoFlow.finding.verdict === "fixed" && getRegression(gone.id) === null); + + // A finding marked real before regressions existed goes straight to active on fixed. + const legacy = storeFinding(sample({ fingerprint: "r-legacy", check: "frozen" }), DRY).finding; + db.prepare("UPDATE findings SET verdict = 'real' WHERE id = ?").run(legacy.id); + const lf = setVerdict(legacy.id, "fixed", null, null, { readArtifact: read }); + check("regressions: fixed on a pre-regression real finding makes an active test", lf.ok && lf.regression?.state === "active" && lf.regression.asserts === "replays"); + + // Filters. + const fixedOnly = listFindings({ verdict: "fixed" }); + check("list: the verdict filter", fixedOnly.ok && fixedOnly.body.findings.every((f) => f.verdict === "fixed") && fixedOnly.body.total === 2, + JSON.stringify(fixedOnly.ok && fixedOnly.body.total)); + check("list: a bad verdict filter is refused", !listFindings({ verdict: "meh" }).ok); + const page2 = listFindings({ limit: 1, offset: 1 }); + check("list: offset pages, total counts every match", page2.ok && page2.body.findings.length === 1 && page2.body.total === 3); + check("list: rows carry their regression state", fixedOnly.ok && fixedOnly.body.findings.some((f) => f.regression?.state === "active")); + check("list: since accepts a span", listFindings({ since: "24h" }).ok && !listFindings({ since: "yesterday-ish" }).ok); + }); + + // --- small parsers --- + { + const now = new Date("2026-10-15T12:00:00Z"); + check("since: 24h is a day back", parseSince("24h", now)?.toISOString() === "2026-10-14T12:00:00.000Z"); + check("since: 7d and 90m", parseSince("7d", now)?.toISOString() === "2026-10-08T12:00:00.000Z" && parseSince("90m", now)?.toISOString() === "2026-10-15T10:30:00.000Z"); + check("since: ISO passes through, nonsense is null", parseSince("2026-10-01", now) !== null && parseSince("soon", now) === null); + const vb = parseVerdictBody({ verdict: "fixed", note: " PR #9 " }); + check("verdict body: fixed and a trimmed note", vb.ok && vb.verdict === "fixed" && vb.note === "PR #9"); + check("verdict body: missing verdict, bad verdict, bad duplicate_of are refused", + !parseVerdictBody({}).ok && !parseVerdictBody({ verdict: "meh" }).ok && !parseVerdictBody({ verdict: "duplicate", duplicate_of: "x" }).ok); + const run = parseRegressionRun({ job_id: "j", device_id: "d", outcome: "reproduced" }); + check("regression run: build defaults to installed", run.ok && run.run.build === "installed"); + check("regression run: an unknown outcome is refused", !parseRegressionRun({ job_id: "j", device_id: "d", outcome: "meh" }).ok); + } + + // --- the migration: a database whose findings table predates `fixed` --- + { + closeDb(); + const dir = mkdtempSync(path.join(tmpdir(), "fleet-findings-migrate-")); + try { + mkdirSync(dir, { recursive: true }); + const seed = new DatabaseSync(path.join(dir, "fleet.db")); + seed.exec(`CREATE TABLE findings ( + id INTEGER PRIMARY KEY AUTOINCREMENT, fingerprint TEXT NOT NULL UNIQUE, app TEXT NOT NULL, build TEXT NOT NULL, + platform TEXT NOT NULL, device_id TEXT NOT NULL, job_id TEXT NOT NULL, last_job_id TEXT, mission_id TEXT NOT NULL, + check_name TEXT NOT NULL, subclass TEXT, severity TEXT NOT NULL CHECK (severity IN ('high','medium','low')), + title TEXT NOT NULL, detail TEXT NOT NULL DEFAULT '', screen TEXT NOT NULL DEFAULT '', screen_name TEXT NOT NULL DEFAULT '', + steps TEXT NOT NULL DEFAULT '[]', replay TEXT, artifacts TEXT NOT NULL DEFAULT '{}', + first_seen TEXT NOT NULL DEFAULT (datetime('now')), last_seen TEXT NOT NULL DEFAULT (datetime('now')), + seen_count INTEGER NOT NULL DEFAULT 1, builds_seen TEXT NOT NULL DEFAULT '[]', + verdict TEXT CHECK (verdict IS NULL OR verdict IN ('real','duplicate','not_a_bug','agent_mistake')), + verdict_note TEXT, verdict_at TEXT, duplicate_of INTEGER, issue TEXT, issue_state TEXT, issue_at TEXT)`); + seed.exec(`INSERT INTO findings (fingerprint, app, build, platform, device_id, job_id, mission_id, check_name, severity, title, verdict, verdict_note, verdict_at) + VALUES ('old-1', 'plants', '1', 'android', 'd', 'j', 'm', 'crash', 'high', 'Old crash', 'real', 'from before', '2026-10-01 08:00:00')`); + seed.close(); + openDb(dir); + const r = setVerdict(1, "fixed", null, null, { readArtifact: () => null }); + check("migration: an old database accepts fixed", r.ok && r.finding.verdict === "fixed", JSON.stringify(r)); + const h = findingDetail(1, () => null)!.history; + check("migration: the verdict given before history existed is in the history", + h.length === 2 && h[0].verdict === "real" && h[0].note === "from before" && h[0].at === "2026-10-01T08:00:00Z", JSON.stringify(h)); + } finally { + closeDb(); + rmSync(dir, { recursive: true, force: true }); + } + } } diff --git a/collector/src/findings.ts b/collector/src/findings.ts index 0d92d49..955e6ad 100644 --- a/collector/src/findings.ts +++ b/collector/src/findings.ts @@ -9,7 +9,11 @@ * * store upsert on the fingerprint, so the fifth night that finds the * same crash raises a count instead of a fifth report (T2) - * verdict a person's real / duplicate / not a bug / agent's mistake (T4) + * verdict a person's real / duplicate / not a bug / agent's mistake (T4), + * and fixed, which only follows real; every one kept in a history + * regressions a real finding's replay flow kept as a candidate test, made + * active when it is marked fixed, and run by the app's nightly + * ui-test; one that reproduces reopens the finding (LOOP Phase 4) * precision per app and check, how often a person called it real -- and * which check classes are wasting the morning (O4) * digest the morning summary, sent once a day to the phone (T5) @@ -36,11 +40,14 @@ * other. Even armed, this code only ever creates issues. It never edits, * comments on or closes one; that is a person's job. */ +import { existsSync, readFileSync, statSync } from "node:fs"; +import path from "node:path"; import { db } from "./db.js"; import { iso, parse } from "./api/shared.js"; import { sendWebhook } from "./alerts.js"; import { skipReasonText } from "./skip.js"; import { + ARTIFACT_DIR, DASH_URL, FINDINGS_DIGEST_AT, FINDINGS_DIGEST_HOURS, @@ -55,7 +62,7 @@ import { CHECK_NAMES, type CheckName, type ExploreFinding, type Verdict } from " // The rules, as numbers // --------------------------------------------------------------------------- -export const VERDICTS: readonly Verdict[] = ["real", "duplicate", "not_a_bug", "agent_mistake"]; +export const VERDICTS: readonly Verdict[] = ["real", "duplicate", "not_a_bug", "agent_mistake", "fixed"]; const SEVERITIES = ["high", "medium", "low"] as const; const REPLAY_KINDS = ["maestro", "tvloop", "none"] as const; const ARTIFACT_KEYS = ["shot", "sheet", "trajectory", "log", "replay"] as const; @@ -314,6 +321,8 @@ export type FindingView = { duplicate_of: number | null; issue: IssueRecord | null; status: "open" | "triaged"; + /** The regression test this finding became, if it did: just enough for a list. */ + regression: { state: RegressionState; kind: RegressionKind; last_outcome: RegressionOutcome | null } | null; }; export function replayLabel(replay: Replay, check: CheckName): string { @@ -354,6 +363,7 @@ function view(r: Row): FindingView { duplicate_of: r.duplicate_of, issue: parse(r.issue, null), status: r.verdict === null ? "open" : "triaged", + regression: regressionBrief(r.id), }; } @@ -636,6 +646,40 @@ export function listIssues(limit = 100) { // Verdicts (T4) // --------------------------------------------------------------------------- +/** Who gave a verdict, for the history. Null fields when the collector cannot say. */ +export type Who = { actor: string | null; source: string | null }; + +export type VerdictOptions = { + who?: Who; + /** Reads a replay flow out of the artifact store. Tests pass their own. */ + readArtifact?: ReadArtifact; +}; + +export type VerdictOutcome = + | { ok: true; finding: FindingView; regression: RegressionView | null; change: string | null } + | ({ ok: false } & { status: 400 | 404 | 409; error: string; hint?: string }); + +/** + * A verdict body as either route receives it: `{verdict, note?, duplicate_of?}`. + * Parsed here so the legacy route and /api/v1 refuse the same things in the + * same words. + */ +export function parseVerdictBody( + body: unknown, +): { ok: true; verdict: Verdict | null; note: string | null; duplicateOf: number | null } | { ok: false; error: string } { + const b = (body && typeof body === "object" && !Array.isArray(body) ? body : {}) as { + verdict?: unknown; note?: unknown; duplicate_of?: unknown; + }; + if (!("verdict" in b)) return { ok: false, error: "verdict is required (or null to reopen)" }; + if (b.verdict !== null && !VERDICTS.includes(b.verdict as Verdict)) + return { ok: false, error: `verdict must be one of ${VERDICTS.join(", ")}, or null to reopen` }; + if (b.note !== undefined && b.note !== null && typeof b.note !== "string") return { ok: false, error: "note must be a string" }; + const dup = b.duplicate_of === undefined || b.duplicate_of === null ? null : Number(b.duplicate_of); + if (dup !== null && !(Number.isInteger(dup) && dup > 0)) return { ok: false, error: "duplicate_of must be a finding id" }; + const note = typeof b.note === "string" && b.note.trim() ? b.note.trim() : null; + return { ok: true, verdict: (b.verdict ?? null) as Verdict | null, note, duplicateOf: dup }; +} + /** * Record a person's verdict, or clear it with `verdict: null` (a mis-click * should not need SQL to undo). @@ -644,13 +688,20 @@ export function listIssues(limit = 100) { * is a duplicate of A and C is marked a duplicate of B, C points at A, so the * "this is the same as" graph never grows chains a person has to walk. Marking * something a duplicate of itself, or of one of its own duplicates, is refused. + * + * `fixed` is refused unless the finding is already `real` (or was, and a + * regression test brought it back): "fixed" on something nobody confirmed is a + * claim about a bug that may not exist, and it would put a test in the nightly + * for it. Every verdict lands in the history and moves the regression test, + * see `moveRegression`. */ -export function setVerdict( +export const setVerdict = db.transaction(( id: number, verdict: Verdict | null, note: string | null, duplicateOf: number | null, -): { ok: true; finding: FindingView } | ({ ok: false } & { status: 400 | 404; error: string }) { + opts: VerdictOptions = {}, +): VerdictOutcome => { const row = getRow(id); if (!row) return { ok: false, status: 404, error: `no finding ${id}` }; if (verdict !== null && !VERDICTS.includes(verdict)) @@ -658,6 +709,14 @@ export function setVerdict( if (note !== null && note.length > LIMITS.note) return { ok: false, status: 400, error: `note is longer than ${LIMITS.note} characters` }; + const reg = getRegressionRow(id); + if (verdict === "fixed" && row.verdict !== "real" && row.verdict !== "fixed" && reg?.state !== "active") + return { + ok: false, status: 409, + error: `finding ${id} is ${row.verdict === null ? "open" : row.verdict}; fixed comes only after real`, + hint: "mark it real first, and fixed once the fix has merged", + }; + let target: number | null = null; if (verdict === "duplicate") { if (duplicateOf === null) return { ok: false, status: 400, error: "a duplicate needs duplicate_of: the id it duplicates" }; @@ -677,17 +736,302 @@ export function setVerdict( duplicate_of = ? WHERE id = ?`, ).run(verdict, verdict === null ? null : note, verdict, target, id); - return { ok: true, finding: view(getRow(id)!) }; + db.prepare("INSERT INTO finding_verdicts (finding_id, event, verdict, note, actor, source) VALUES (?, 'verdict', ?, ?, ?, ?)") + .run(id, verdict, note, opts.who?.actor ?? null, opts.who?.source ?? null); + const change = moveRegression(view(getRow(id)!), verdict, opts.readArtifact ?? readStoredArtifact); + return { ok: true, finding: view(getRow(id)!), regression: regressionView(getRegressionRow(id)), change }; +}); + +export type HistoryEntry = { + at: string | null; + /** `verdict`: a person's call (verdict null = reopened). `regressed`: a regression test reproduced it. */ + event: "verdict" | "regressed"; + verdict: Verdict | null; + note: string | null; + actor: string | null; + source: string | null; +}; + +export function verdictHistory(id: number): HistoryEntry[] { + const rows = db + .prepare("SELECT at, event, verdict, note, actor, source FROM finding_verdicts WHERE finding_id = ? ORDER BY id") + .all(id) as (Omit & { at: string })[]; + return rows.map((r) => ({ ...r, at: iso(r.at) })); +} + +// --------------------------------------------------------------------------- +// Regressions: a fixed finding becomes a test (LOOP Phase 4) +// --------------------------------------------------------------------------- + +export type RegressionState = "candidate" | "active" | "retired"; +export type RegressionKind = "maestro" | "tvloop"; +/** + * One night's run of a regression flow on one device: + * passed the flow replayed without error (and, for a crash or ANR, the + * package did not crash while it ran) + * reproduced the original failure came back: a crash or ANR of the package + * during the flow. Reopens the finding. + * failed the flow itself failed. For any check but crash and anr this + * is all a replay can say: the steps no longer replay, which may + * be the bug or may be a changed screen. Does not reopen. + * skipped not run on this device (wrong platform, no Maestro there) + */ +export type RegressionOutcome = "passed" | "reproduced" | "failed" | "skipped"; +export const REGRESSION_OUTCOMES: readonly RegressionOutcome[] = ["passed", "reproduced", "failed", "skipped"]; + +/** Checks a replay can assert the absence of. Everything else is "replays without error". */ +export const ABSENCE_CHECKS: readonly CheckName[] = ["crash", "anr"]; + +/** A flow bigger than this is not a flow a person recorded; it is not kept. */ +export const FLOW_MAX_BYTES = 64 * 1024; + +type RegressionRow = { + finding_id: number; + app: string; + platform: string; + check_name: CheckName; + kind: RegressionKind; + flow: string; + flow_sha256: string | null; + state: RegressionState; + created_at: string; + activated_at: string | null; + retired_at: string | null; + runs: number; + last_run_at: string | null; + last_job_id: string | null; + last_outcome: RegressionOutcome | null; + last_detail: string | null; +}; + +export type RegressionView = { + finding_id: number; + app: string; + platform: string; + check: CheckName; + kind: RegressionKind; + state: RegressionState; + /** What a run proves: `no_crash` for crash and anr findings, `replays` (without error) for the rest. */ + asserts: "no_crash" | "replays"; + flow: string; + flow_sha256: string | null; + created_at: string | null; + activated_at: string | null; + retired_at: string | null; + runs: number; + last_run_at: string | null; + last_job_id: string | null; + last_outcome: RegressionOutcome | null; + last_detail: string | null; +}; + +/** The flow text for a sha, or null when the store does not have it (or it is too big to be a flow). */ +export type ReadArtifact = (sha256: string) => string | null; + +export const readStoredArtifact: ReadArtifact = (sha) => { + if (!SHA.test(sha)) return null; + const file = path.join(ARTIFACT_DIR, sha); + try { + if (statSync(file).size > FLOW_MAX_BYTES) return null; + return readFileSync(file, "utf8"); + } catch { + return null; + } +}; + +const getRegressionRow = (id: number) => + db.prepare("SELECT * FROM regressions WHERE finding_id = ?").get(id) as RegressionRow | undefined; + +function regressionView(r: RegressionRow | undefined): RegressionView | null { + if (!r) return null; + return { + finding_id: r.finding_id, app: r.app, platform: r.platform, check: r.check_name, kind: r.kind, state: r.state, + asserts: ABSENCE_CHECKS.includes(r.check_name) ? "no_crash" : "replays", + flow: r.flow, flow_sha256: r.flow_sha256, + created_at: iso(r.created_at), activated_at: iso(r.activated_at), retired_at: iso(r.retired_at), + runs: r.runs, last_run_at: iso(r.last_run_at), last_job_id: r.last_job_id, last_outcome: r.last_outcome, last_detail: r.last_detail, + }; +} + +function regressionBrief(id: number): FindingView["regression"] { + const r = db.prepare("SELECT state, kind, last_outcome FROM regressions WHERE finding_id = ?").get(id) as + | { state: RegressionState; kind: RegressionKind; last_outcome: RegressionOutcome | null } + | undefined; + return r ? { state: r.state, kind: r.kind, last_outcome: r.last_outcome } : null; +} + +export function getRegression(id: number): RegressionView | null { + return regressionView(getRegressionRow(id)); +} + +/** The replay flow's artifact: the replay's own hash, or the replay artifact a crash posts without counts. */ +const flowShaOf = (f: Pick) => f.replay?.sha256 ?? f.artifacts.replay ?? null; + +/** + * Copy the finding's replay flow into the regressions table. Returns why not, + * when it cannot: no replay file was posted, or the store no longer has it. + */ +function keepFlow(f: FindingView, state: RegressionState, read: ReadArtifact): string { + const sha = flowShaOf(f); + if (!sha) return "no replay flow was posted with this finding, so there is nothing to keep as a test"; + if (f.replay?.kind === "none") return "the replay was not a file (kind none), so there is nothing to keep as a test"; + const text = read(sha); + if (text === null) return `the replay flow ${sha.slice(0, 12)} is not in the artifact store (collected, or over ${FLOW_MAX_BYTES} bytes)`; + const kind: RegressionKind = + f.replay?.kind === "tvloop" || f.replay?.kind === "maestro" ? f.replay.kind : text.trimStart().startsWith("{") ? "tvloop" : "maestro"; + db.prepare( + `INSERT INTO regressions (finding_id, app, platform, check_name, kind, flow, flow_sha256, state, activated_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, CASE WHEN ? = 'active' THEN datetime('now') END)`, + ).run(f.id, f.app, f.platform, f.check, kind, text, sha, state, state); + return state === "active" + ? `its replay flow is now an active regression test in ${f.app}'s nightly ui-test` + : `its replay flow is kept as a candidate regression test; it joins the nightly when the finding is marked fixed`; } +/** + * What a verdict does to the finding's regression test. + * + * real keep the replay flow as a candidate (a retired one + * comes back as a candidate; an active one stays + * active -- it is watching for this very bug) + * fixed candidate -> active: the app's nightly runs it + * duplicate, not_a_bug, + * agent_mistake retire it: it is not a test of anything real + * null (reopen) leave it as it is + * + * Returns one sentence saying what changed, or null when nothing did. + */ +function moveRegression(f: FindingView, verdict: Verdict | null, read: ReadArtifact): string | null { + const reg = getRegressionRow(f.id); + if (verdict === "real") { + if (!reg) return keepFlow(f, "candidate", read); + if (reg.state === "retired") { + db.prepare("UPDATE regressions SET state = 'candidate', retired_at = NULL WHERE finding_id = ?").run(f.id); + return "its retired regression test is a candidate again"; + } + return null; + } + if (verdict === "fixed") { + if (!reg) return keepFlow(f, "active", read); + if (reg.state === "active") return null; + db.prepare("UPDATE regressions SET state = 'active', activated_at = datetime('now'), retired_at = NULL WHERE finding_id = ?").run(f.id); + return `its replay flow is now an active regression test in ${f.app}'s nightly ui-test`; + } + if (verdict !== null && reg && reg.state !== "retired") { + db.prepare("UPDATE regressions SET state = 'retired', retired_at = datetime('now') WHERE finding_id = ?").run(f.id); + return `its ${reg.state} regression test is retired`; + } + return null; +} + +/** What a nightly ui-test pulls at run time: the app's regressions in one state (active by default). */ +export function listRegressions(q: { app?: string; state?: string; platform?: string } = {}): { ok: true; items: RegressionView[] } | { ok: false; error: string } { + const state = q.state ?? "active"; + if (state !== "all" && !["candidate", "active", "retired"].includes(state)) + return { ok: false, error: "state must be candidate, active, retired or all" }; + const where: string[] = []; + const params: string[] = []; + if (q.app) { where.push("app = ?"); params.push(q.app); } + if (q.platform) { where.push("platform = ?"); params.push(q.platform); } + if (state !== "all") { where.push("state = ?"); params.push(state); } + const rows = db + .prepare(`SELECT * FROM regressions ${where.length ? `WHERE ${where.join(" AND ")}` : ""} ORDER BY finding_id`) + .all(...params) as RegressionRow[]; + return { ok: true, items: rows.map((r) => regressionView(r)!) }; +} + +export type RegressionRun = { job_id: string; device_id: string; build: string; outcome: RegressionOutcome; detail: string }; + +/** A run report as the executor posts it, checked. */ +export function parseRegressionRun(body: unknown): { ok: true; run: RegressionRun } | { ok: false; error: string } { + if (!body || typeof body !== "object" || Array.isArray(body)) return { ok: false, error: "body must be a JSON object" }; + const b = body as Record; + for (const k of ["job_id", "device_id"]) { + if (typeof b[k] !== "string" || !(b[k] as string).trim() || (b[k] as string).length > LIMITS.id) + return { ok: false, error: `${k} is required (a non-empty string of at most ${LIMITS.id} characters)` }; + } + if (b.build !== undefined && b.build !== null && (typeof b.build !== "string" || b.build.length > LIMITS.id)) + return { ok: false, error: `build must be a string of at most ${LIMITS.id} characters` }; + if (!REGRESSION_OUTCOMES.includes(b.outcome as RegressionOutcome)) + return { ok: false, error: `outcome must be one of ${REGRESSION_OUTCOMES.join(", ")}` }; + if (b.detail !== undefined && b.detail !== null && typeof b.detail !== "string") return { ok: false, error: "detail must be a string" }; + return { + ok: true, + run: { + job_id: b.job_id as string, device_id: b.device_id as string, + build: (typeof b.build === "string" && b.build.trim()) || "installed", + outcome: b.outcome as RegressionOutcome, + detail: typeof b.detail === "string" ? b.detail.slice(0, LIMITS.note) : "", + }, + }; +} + +/** + * Record one device's run of a regression flow. + * + * `reproduced` on an ACTIVE regression reopens the finding -- verdict cleared, + * seen once more, the build added, and a history line saying which build it + * came back on -- so the morning digest lists it again instead of the nightly + * passing over it. A fan-out reporting the same reproduction from three + * devices of one job reopens it once. The regression stays active: the bug is + * back, and the nightly should keep saying so until someone marks it fixed + * again (real, then fixed). A candidate or retired regression's run is + * recorded and changes nothing else, since nothing schedules those. + */ +export const recordRegressionRun = db.transaction((id: number, run: RegressionRun): + | { ok: true; reopened: boolean; regression: RegressionView; finding: FindingView } + | { ok: false; status: 404; error: string } => { + const reg = getRegressionRow(id); + const row = getRow(id); + if (!reg || !row) return { ok: false, status: 404, error: `finding ${id} has no regression test` }; + const alreadyThisJob = reg.last_job_id === run.job_id && reg.last_outcome === "reproduced"; + db.prepare( + `UPDATE regressions SET runs = runs + 1, last_run_at = datetime('now'), last_job_id = ?, last_outcome = ?, last_detail = ? + WHERE finding_id = ?`, + ).run(run.job_id, alreadyThisJob ? "reproduced" : run.outcome, run.detail || null, id); + + let reopened = false; + if (run.outcome === "reproduced" && reg.state === "active" && !alreadyThisJob) { + const builds = parse(row.builds_seen, []); + if (!builds.includes(run.build)) builds.push(run.build); + db.prepare( + `UPDATE findings SET verdict = NULL, verdict_note = NULL, verdict_at = NULL, duplicate_of = NULL, + seen_count = seen_count + 1, last_seen = datetime('now'), last_job_id = ?, builds_seen = ? + WHERE id = ?`, + ).run(run.job_id, JSON.stringify(builds), id); + const note = `regressed on build ${run.build} (job ${run.job_id}, ${run.device_id})${run.detail ? `: ${run.detail}` : ""}`.slice(0, LIMITS.note); + db.prepare("INSERT INTO finding_verdicts (finding_id, event, verdict, note, source) VALUES (?, 'regressed', NULL, ?, 'regression')") + .run(id, note); + reopened = true; + } + return { ok: true, reopened, regression: regressionView(getRegressionRow(id))!, finding: view(getRow(id)!) }; +}); + // --------------------------------------------------------------------------- // Reading // --------------------------------------------------------------------------- -export type ListQuery = { app?: string; check?: string; status?: string; since?: string; limit?: number }; +export type ListQuery = { + app?: string; check?: string; status?: string; verdict?: string; since?: string; limit?: number; offset?: number; +}; + +/** + * A `since` filter: an ISO-8601 time, or a span back from now -- `90m`, `24h`, + * `7d`, `2w` -- which is what an agent's morning brief asks for. Null when it + * is neither. + */ +export function parseSince(raw: string, now = new Date()): Date | null { + const span = /^(\d+)\s*(m|h|d|w)$/i.exec(raw.trim()); + if (span) { + const unit = { m: 60_000, h: 3_600_000, d: 86_400_000, w: 604_800_000 }[span[2].toLowerCase() as "m" | "h" | "d" | "w"]; + return new Date(now.getTime() - Number(span[1]) * unit); + } + const t = Date.parse(raw); + return Number.isFinite(t) ? new Date(t) : null; +} /** The filtered list, newest sighting first, with facets over every row so a filter never empties its own menu. */ -export function listFindings(q: ListQuery): { ok: true; body: unknown } | { ok: false; error: string } { +export function listFindings(q: ListQuery): { ok: true; body: FindingList } | { ok: false; error: string } { const where: string[] = []; const params: string[] = []; if (q.app) { where.push("app = ?"); params.push(q.app); } @@ -695,13 +1039,13 @@ export function listFindings(q: ListQuery): { ok: true; body: unknown } | { ok: const status = q.status ?? "all"; if (!["open", "triaged", "all"].includes(status)) return { ok: false, error: "status must be open, triaged or all" }; if (q.since) { - const t = Date.parse(q.since); - if (!Number.isFinite(t)) return { ok: false, error: "since must be an ISO-8601 date or time" }; + const t = parseSince(q.since); + if (!t) return { ok: false, error: "since must be an ISO-8601 date or time, or a span like 24h or 7d" }; where.push("last_seen >= ?"); - params.push(sqliteTime(new Date(t))); + params.push(sqliteTime(t)); } - // Counted before the status filter, so the page can say "3 open, 12 triaged" - // for whatever app and check are selected. + // Counted before the status and verdict filters, so the page can say + // "3 open, 12 triaged" for whatever app and check are selected. const scoped = where.length ? `WHERE ${where.join(" AND ")}` : ""; const counts = db .prepare( @@ -711,13 +1055,20 @@ export function listFindings(q: ListQuery): { ok: true; body: unknown } | { ok: .get(...params) as { open: number | null; triaged: number | null; total: number }; if (status === "open") where.push("verdict IS NULL"); if (status === "triaged") where.push("verdict IS NOT NULL"); + if (q.verdict) { + const wanted = q.verdict.split(",").map((v) => v.trim()).filter(Boolean); + const bad = wanted.find((v) => !VERDICTS.includes(v as Verdict)); + if (bad) return { ok: false, error: `verdict must be one of ${VERDICTS.join(", ")} (comma-separated), not ${JSON.stringify(bad)}` }; + where.push(`verdict IN (${wanted.map(() => "?").join(",")})`); + params.push(...wanted); + } + const filtered = where.length ? `WHERE ${where.join(" AND ")}` : ""; + const total = (db.prepare(`SELECT COUNT(*) AS n FROM findings ${filtered}`).get(...params) as { n: number }).n; const limit = Math.min(500, Math.max(1, Number(q.limit ?? 100) || 100)); + const offset = Math.max(0, Math.floor(Number(q.offset ?? 0) || 0)); const rows = db - .prepare( - `SELECT * FROM findings ${where.length ? `WHERE ${where.join(" AND ")}` : ""} - ORDER BY last_seen DESC, id DESC LIMIT ?`, - ) - .all(...params, limit) as Row[]; + .prepare(`SELECT * FROM findings ${filtered} ORDER BY last_seen DESC, id DESC LIMIT ? OFFSET ?`) + .all(...params, limit, offset) as Row[]; return { ok: true, body: { @@ -727,22 +1078,80 @@ export function listFindings(q: ListQuery): { ok: true; body: unknown } | { ok: checks: (db.prepare("SELECT DISTINCT check_name FROM findings ORDER BY check_name").all() as { check_name: string }[]) .map((r) => r.check_name), limit, + offset, + total, }, }; } -/** One finding, with the findings a person has marked as duplicates of it. */ -export function findingDetail(id: number) { +export type FindingList = { + findings: FindingView[]; + counts: { open: number; triaged: number; total: number }; + apps: string[]; + checks: string[]; + limit: number; + offset: number; + /** Rows matching every filter, status and verdict included. */ + total: number; +}; + +const EVIDENCE_LABEL = ARTIFACT_LABEL; + +/** A replay flow small enough to inline is inlined; past this, follow `url`. */ +export const FLOW_INLINE_BYTES = 16 * 1024; + +/** + * One finding, with the findings a person has marked as duplicates of it, its + * verdict history, its regression test, its evidence as links, and its replay + * flow -- inline when it is small, which a Maestro flow nearly always is. + */ +export function findingDetail(id: number, read: ReadArtifact = readStoredArtifact) { const f = getFinding(id); if (!f) return null; const duplicates = db .prepare("SELECT id, title, seen_count FROM findings WHERE duplicate_of = ? ORDER BY id") .all(id) as { id: number; title: string; seen_count: number }[]; const original = f.duplicate_of ? getFinding(f.duplicate_of) : null; + const inStore = (sha: string) => existsSync(path.join(ARTIFACT_DIR, sha)); + const evidence = ARTIFACT_KEYS.filter((k) => f.artifacts[k]).map((k) => { + const sha = f.artifacts[k]!; + return { role: k, label: EVIDENCE_LABEL[k], sha256: sha, url: `/api/v1/artifacts/${sha}/content`, in_store: inStore(sha) }; + }); + const regression = getRegression(id); + const flowSha = flowShaOf(f); + let replayFlow: { + kind: string; sha256: string; url: string; in_store: boolean; text: string | null; truncated: boolean; + } | null = null; + if (flowSha) { + const stored = inStore(flowSha); + const text = (stored ? read(flowSha) : null) ?? (regression?.flow_sha256 === flowSha ? regression.flow : null); + const kind = f.replay?.kind && f.replay.kind !== "none" ? f.replay.kind : regression?.kind ?? "maestro"; + replayFlow = { + kind, sha256: flowSha, url: `/api/v1/artifacts/${flowSha}/content`, in_store: stored, + text: text !== null && Buffer.byteLength(text) <= FLOW_INLINE_BYTES ? text : null, + truncated: text !== null && Buffer.byteLength(text) > FLOW_INLINE_BYTES, + }; + } return { ...f, duplicates, duplicate_of_title: original?.title ?? null, + counts: { + seen: f.seen_count, + builds: f.builds_seen.length, + replay_attempts: f.replay?.attempts ?? null, + replay_reproduced: f.replay?.reproduced ?? null, + regression_runs: regression?.runs ?? 0, + }, + evidence, + replay_flow: replayFlow, + regression, + history: verdictHistory(id), + links: { + self: `/api/v1/findings/${id}`, + verdict: `/api/v1/findings/${id}/verdict`, + dashboard: `/dash/findings/${id}`, + }, }; } @@ -760,7 +1169,7 @@ export type PrecisionClass = { duplicate: number; not_a_bug: number; agent_mistake: number; - /** real + not_a_bug + agent_mistake. Duplicates say nothing about whether the check was right. */ + /** real (fixed included) + not_a_bug + agent_mistake. Duplicates say nothing about whether the check was right. */ judged: number; /** real / judged, or null before anything has been judged. */ precision: number | null; @@ -793,7 +1202,8 @@ export function precision(opts: { app?: string; min?: number } = {}) { c = { app: r.app, check: r.check_name, subclass: r.subclass || null, open: 0, real: 0, duplicate: 0, not_a_bug: 0, agent_mistake: 0, judged: 0, precision: null, disabled: false }; byKey.set(key, c); } - c[r.verdict ?? "open"] += r.n; + // A fixed finding was a real one: the check was right about it. + c[r.verdict === "fixed" ? "real" : r.verdict ?? "open"] += r.n; } const classes = [...byKey.values()].map((c) => { const judged = c.real + c.not_a_bug + c.agent_mistake; diff --git a/collector/src/workloads/explore/types.ts b/collector/src/workloads/explore/types.ts index 5997a39..7e0e8cc 100644 --- a/collector/src/workloads/explore/types.ts +++ b/collector/src/workloads/explore/types.ts @@ -305,5 +305,9 @@ export type ExploreFinding = { artifacts: { shot?: string; sheet?: string; trajectory?: string; log?: string; replay?: string }; }; -/** A person's verdict on a finding (T4). Also the training label if E8 ever happens. */ -export type Verdict = "real" | "duplicate" | "not_a_bug" | "agent_mistake"; +/** + * A person's verdict on a finding (T4). Also the training label if E8 ever happens. + * `fixed` comes only after `real`: the bug was real and a fix has merged, which + * is what makes its replay flow a regression test in the nightly. + */ +export type Verdict = "real" | "duplicate" | "not_a_bug" | "agent_mistake" | "fixed"; diff --git a/collector/src/workloads/regressions.test.ts b/collector/src/workloads/regressions.test.ts new file mode 100644 index 0000000..5d419df --- /dev/null +++ b/collector/src/workloads/regressions.test.ts @@ -0,0 +1,91 @@ +/** + * The ui-test side of regression tests, with no device: which app a job asks + * for, how a stored flow is made runnable, what a run is judged, and what is + * reported -- against fake crash readings and a fake Maestro. + * + * Run from scripts/smoke.ts like the other decision checks. What it cannot + * show is Maestro and adb themselves, which only a device can. + */ +import path from "node:path"; +import { flowAppId, judgeRegression, prepareFlow, regressionApp, runRegressions, type CrashReading, type RegressionDeps } from "./regressions.js"; +import type { Target } from "./types.js"; + +type Check = (name: string, cond: boolean, detail?: string) => void; + +const throws = (fn: () => unknown) => { + try { + fn(); + return false; + } catch { + return true; + } +}; + +export async function runRegressionChecks(check: Check) { + // Which app. + check("regressions: no param, no regressions", regressionApp({ params: {} }) === null); + check("regressions: a name is the app", regressionApp({ params: { regressions: "greenfolio" } }) === "greenfolio"); + check("regressions: true is job.app.name", + regressionApp({ params: { regressions: true }, app: { name: "greenfolio-android", build: "1", sha256: "latest" } }) === "greenfolio-android"); + check("regressions: true with no app, or a nonsense value, fails the job", + throws(() => regressionApp({ params: { regressions: true } })) && throws(() => regressionApp({ params: { regressions: 7 } }))); + + // Making a stored flow runnable. + const root = path.resolve("/lab/flows"); + const flow = 'appId: com.example.app.debug\nname: "x"\n---\n- launchApp:\n clearState: true\n- runFlow: "greenfolio/login.yaml"\n- tapOn:\n point: "50%,50%"\n'; + const ready = prepareFlow(flow, { appId: "com.example.app.smoke", flowsDir: root }); + check("prepare: the job's appId replaces the recorded one", /^appId: com\.example\.app\.smoke$/m.test(ready) && !ready.includes(".debug"), ready); + check("prepare: a relative runFlow is resolved under the flows dir", + ready.includes(`- runFlow: ${JSON.stringify(path.join(root, "greenfolio/login.yaml"))}`), ready); + check("prepare: without an appId the flow keeps its own", flowAppId(prepareFlow(flow, { flowsDir: root })) === "com.example.app.debug"); + check("prepare: a runFlow escaping the flows dir is refused", + throws(() => prepareFlow('appId: a\n---\n- runFlow: "../../etc/x.yaml"\n', { flowsDir: root }))); + check("prepare: a parameterised appId is not a package", flowAppId("appId: ${APP_ID}\n---\n") === null); + + // Judging. + const none = { count: 0, signatures: [], problem: null }; + const one = { count: 1, signatures: ["java.lang.NullPointerException: boom"], problem: null }; + const r1 = judgeRegression({ check: "crash", platform: "android", flowError: "Element not found", before: none, after: one }); + check("judge: a new crash during a crash finding's flow is reproduced", r1.outcome === "reproduced" && r1.detail.includes("NullPointerException"), r1.detail); + const r2 = judgeRegression({ check: "anr", platform: "android", flowError: null, before: one, after: one }); + check("judge: no new crash and a clean flow passes", r2.outcome === "passed"); + const r3 = judgeRegression({ check: "crash", platform: "android", flowError: "Assertion failed", before: none, after: none }); + check("judge: a failing flow with no crash is failed, not reproduced", r3.outcome === "failed" && /no crash/.test(r3.detail)); + const r4 = judgeRegression({ check: "crash", platform: "ios", flowError: null, before: null, after: null }); + check("judge: on iOS a crash finding's clean replay passes and says the crash was not checked", r4.outcome === "passed" && /not checked/.test(r4.detail), r4.detail); + const r5 = judgeRegression({ check: "dead_control", platform: "android", flowError: "Element not found", before: none, after: none }); + check("judge: other checks never reproduce; a failing flow is failed with the honest caveat", r5.outcome === "failed" && /cannot tell/.test(r5.detail)); + const r6 = judgeRegression({ check: "visual", platform: "android", flowError: null, before: none, after: one }); + check("judge: a crash during a visual finding's flow does not count as that finding", r6.outcome === "passed" && /replayed without error/.test(r6.detail)); + const r7 = judgeRegression({ check: "crash", platform: "android", flowError: null, before: { count: 0, signatures: [], problem: "logcat could not be read" }, after: none }); + check("judge: an unreadable crash log is said, not guessed", r7.outcome === "passed" && /logcat could not be read/.test(r7.detail)); + + // Running: the right flows, the readings around each, and a report for each. + const android: Target = { id: "pixel", platform: "android", kind: "device" } as Target; + const crashes: CrashReading[] = [none, one, one, one]; + const ran: { file: string; env: Record }[] = []; + const reports: { id: number; body: Record }[] = []; + const deps: RegressionDeps = { + runFlow: async (_t, file, _cwd, env) => { ran.push({ file, env }); return ran.length === 2 ? "maestro failed: password=hunter2 Element not found" : null; }, + readCrashes: async () => crashes.shift() ?? one, + report: async (id, body) => { reports.push({ id, body }); }, + }; + const out = await runRegressions({ + job: { job_id: "nightly-1", app: { name: "app", build: "2.0 (7)", sha256: "x" } }, target: android, appId: "com.example.app", + env: { TEST_PASSWORD: "hunter2" }, secrets: ["hunter2"], flowsDir: root, + flows: [ + { finding_id: 1, app: "app", platform: "android", check: "crash", kind: "maestro", flow: "appId: com.example.app\n---\n- launchApp\n" }, + { finding_id: 2, app: "app", platform: "android", check: "goal", kind: "maestro", flow: "appId: com.example.app\n---\n- launchApp\n" }, + { finding_id: 3, app: "app", platform: "ios", check: "crash", kind: "maestro", flow: "appId: com.example.app\n---\n" }, + { finding_id: 4, app: "app", platform: "android", check: "crash", kind: "tvloop", flow: "{}" }, + ], + }, deps); + check("run: only this platform's flows, in order", out.map((r) => r.finding_id).join(",") === "1,2,4", JSON.stringify(out)); + check("run: the crash during flow 1 is reproduced", out[0].outcome === "reproduced"); + check("run: the goal flow that failed is failed, and the password is redacted", out[1].outcome === "failed" && !out[1].detail.includes("hunter2"), out[1].detail); + check("run: a tvloop flow is skipped, not run", out[2].outcome === "skipped" && ran.length === 2); + check("run: the flow gets APP_ID and the sign-in env", ran[0].env.APP_ID === "com.example.app" && ran[0].env.TEST_PASSWORD === "hunter2"); + check("run: each is reported with the job, device and build", + reports.length === 3 && reports[0].id === 1 && reports[0].body.outcome === "reproduced" && reports[0].body.build === "2.0 (7)" + && reports[0].body.device_id === "pixel" && reports[0].body.job_id === "nightly-1", JSON.stringify(reports[0])); +} diff --git a/collector/src/workloads/regressions.ts b/collector/src/workloads/regressions.ts new file mode 100644 index 0000000..5331089 --- /dev/null +++ b/collector/src/workloads/regressions.ts @@ -0,0 +1,220 @@ +/** + * Regression tests from fixed findings, as the ui-test workload runs them. + * + * Night QA files a finding with a Maestro flow that replays its steps. A + * person marks it real (the collector keeps the flow as a candidate), then + * fixed once a fix has merged (the flow becomes active). A ui-test job with + * `params.regressions` asks the collector for the app's active flows at run + * time and runs each on every device it tests, after the suite's own flow: + * + * params.regressions: "greenfolio" the explore app key the findings are filed under + * params.regressions: true the same as job.app.name + * + * Pulled at run time rather than copied into the lab's flows directory + * because that directory is outside git: a flow written there by the + * collector would be a file nobody reviews and nothing backs up, on one + * machine. The flow text lives in the collector's database instead, and a + * nightly that already exists picks up every newly fixed finding without + * anyone editing its schedule. + * + * ## What a run can assert + * + * A replay flow repeats the steps that led to the bug. For a crash or an ANR + * that is enough: on Android the crash buffer (`logcat -b crash`, and + * `am_anr` in the events buffer) is read before and after the flow, and a new + * crash of the package during it is the bug coming back -- `reproduced`, + * which reopens the finding. For every other check (frozen, dead_control, + * goal, visual...) the oracle that found it is a model or a heuristic the + * flow does not carry, so the honest claim is weaker: the flow replays + * without error. A flow that fails is reported `failed` and fails the device's + * row, but does not reopen anything, because a step that no longer replays + * may be the bug or may be a redesigned screen. On iOS there is no crash + * buffer read here, so a crash finding's flow is also "replays without + * error" there, and the result says so. + * + * tvloop flows (a Roku's) are kept by the collector but not run by ui-test, + * which drives Maestro; they are reported `skipped`. + */ +import { mkdtempSync, writeFileSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { BASE, exec, post } from "../fleet-client.js"; +import { parseCrashLogcat } from "../soak-samples.js"; +import { redact } from "../secrets.js"; +import { parseAnrEvents } from "./explore/actuators/android.js"; +import { ADB } from "./device.js"; +import { FLOWS_DIR, runFlow } from "./flows.js"; +import type { Job, Target } from "./types.js"; + +/** One regression as `GET /findings/regressions` returns it (the fields this needs). */ +export type RegressionFlow = { + finding_id: number; + app: string; + platform: string; + check: string; + kind: "maestro" | "tvloop"; + flow: string; +}; + +export type RegressionResult = { + finding_id: number; + outcome: "passed" | "reproduced" | "failed" | "skipped"; + detail: string; +}; + +/** Checks whose absence a replay can show: the crash buffer says whether it happened again. */ +const ABSENCE = new Set(["crash", "anr"]); + +/** + * Which app's regressions the job asks for, or null when it asks for none. + * Throws on a value that cannot name one, so a typo fails the job instead of + * quietly running nothing. + */ +export function regressionApp(job: Pick): string | null { + const v = job.params?.regressions; + if (v === undefined || v === false || v === null) return null; + if (v === true) { + if (!job.app?.name) throw new Error("params.regressions: true needs job.app.name; or name the app: params.regressions: \"\""); + return job.app.name; + } + if (typeof v === "string" && /^[\w.@:+-]{1,200}$/.test(v)) return v; + throw new Error("params.regressions must be true or the app name its findings are filed under"); +} + +/** The appId a flow declares, if it declares a literal one. */ +export function flowAppId(flow: string): string | null { + const m = /^appId:\s*(\S+)/m.exec(flow); + return m && !/\$\{/.test(m[1]) ? m[1] : null; +} + +/** + * Make a stored flow runnable here. + * + * - `appId` is replaced by the job's suite.app_id when it sets one: the flow + * was recorded against whatever variant night QA explored (often `.debug`), + * and the nightly may be testing another. + * - `runFlow: "relative/path.yaml"` (the mission's sign-in flow) is resolved + * against the flows directory, because the stored flow is written to a temp + * file and Maestro resolves a relative runFlow against the file it is in. A + * path that escapes the flows directory is refused, as resolveFlow does for + * job specs. + */ +export function prepareFlow(flow: string, opts: { appId?: string | null; flowsDir?: string } = {}): string { + const root = path.resolve(opts.flowsDir ?? FLOWS_DIR); + let out = flow; + if (opts.appId) out = out.replace(/^appId:\s*\S+/m, `appId: ${opts.appId}`); + return out.replace(/^(\s*-\s*runFlow:\s*)(["']?)([^"'\n]+?)\2\s*$/gm, (_m, lead: string, _q: string, rel: string) => { + if (path.isAbsolute(rel)) return `${lead}${JSON.stringify(rel)}`; + const abs = path.resolve(root, rel); + if (abs !== root && !abs.startsWith(root + path.sep)) throw new Error(`the flow's runFlow ${rel} escapes the flows dir`); + return `${lead}${JSON.stringify(abs)}`; + }); +} + +/** Crashes and ANRs of `pkg` in the device's buffers so far; null where nothing here can read them. */ +export type CrashReading = { count: number; signatures: string[]; problem: string | null } | null; + +/** + * Decide one run. Pure: everything it needs is passed in. + * + * `before`/`after` are crash readings around the flow; null means the + * platform has no crash log this reads (anything but Android). + */ +export function judgeRegression(o: { + check: string; platform: string; flowError: string | null; before: CrashReading; after: CrashReading; +}): Omit { + const flowFailed = o.flowError ? `the flow failed: ${o.flowError.split("\n").filter(Boolean).slice(-2).join(" ").slice(0, 300)}` : null; + if (ABSENCE.has(o.check)) { + if (o.before && o.after && !o.before.problem && !o.after.problem) { + const fresh = Math.max(0, o.after.count - o.before.count); + if (fresh > 0) { + const sig = o.after.signatures.slice(-fresh)[0] ?? "crash"; + return { outcome: "reproduced", detail: `the app crashed again during the flow: ${sig}` }; + } + return flowFailed ? { outcome: "failed", detail: `${flowFailed} (no crash of the app during it)` } : { outcome: "passed", detail: "no crash of the app during the flow" }; + } + const why = o.before?.problem ?? o.after?.problem ?? `no crash log is read on ${o.platform}`; + return flowFailed + ? { outcome: "failed", detail: `${flowFailed}; crash not checked (${why})` } + : { outcome: "passed", detail: `replayed without error; crash not checked (${why})` }; + } + return flowFailed + ? { outcome: "failed", detail: `${flowFailed}. A ${o.check} finding's flow cannot tell the bug from a changed screen; look before reopening it` } + : { outcome: "passed", detail: `replayed without error (a ${o.check} finding's flow cannot assert the bug's absence, only that the steps still run)` }; +} + +/** The Android crash and ANR count for `pkg`, the way the explore actuator reads it. */ +export async function readCrashes(target: Target, pkg: string): Promise { + if (target.platform !== "android") return null; + try { + const crash = await exec(ADB, ["-s", target.id, "logcat", "-b", "crash", "-d", "-t", "4000"], { timeout: 30_000, maxBuffer: 16 * 1024 * 1024 }); + const c = parseCrashLogcat(crash.stdout, pkg); + if (c.problem) return { count: 0, signatures: [], problem: c.problem }; + const events = await exec(ADB, ["-s", target.id, "logcat", "-b", "events", "-d", "-t", "4000"], { timeout: 30_000, maxBuffer: 16 * 1024 * 1024 }) + .then((r) => parseAnrEvents(r.stdout, pkg)) + .catch(() => [] as string[]); + return { count: c.count + events.length, signatures: [...c.signatures, ...events], problem: null }; + } catch (e) { + return { count: 0, signatures: [], problem: `logcat could not be read: ${(e as Error).message.slice(0, 120)}` }; + } +} + +/** The app's active regression flows, from the collector. */ +export async function fetchRegressions(app: string): Promise { + const res = await fetch(`${BASE}/findings/regressions?app=${encodeURIComponent(app)}&state=active`); + if (!res.ok) throw new Error(`GET /findings/regressions -> ${res.status}`); + return ((await res.json()) as { regressions: RegressionFlow[] }).regressions; +} + +export type RegressionDeps = { + runFlow: (t: Target, file: string, cwd: string, env: Record, timeoutMs: number) => Promise; + readCrashes: (t: Target, pkg: string) => Promise; + report: (findingId: number, body: Record) => Promise; +}; + +const LIVE: RegressionDeps = { + runFlow, + readCrashes, + report: (id, body) => post(`/findings/${id}/regression`, body), +}; + +/** + * Run the regressions that apply to this target, report each to the + * collector, and return them for the device's result row. + */ +export async function runRegressions(o: { + job: Pick; target: Target; flows: RegressionFlow[]; + appId?: string | null; env?: Record; secrets?: string[]; flowsDir?: string; timeoutMs?: number; +}, deps: RegressionDeps = LIVE): Promise { + const out: RegressionResult[] = []; + const build = o.job.app?.build ?? "installed"; + const mine = o.flows.filter((f) => f.platform === o.target.platform); + if (!mine.length) return out; + const dir = mkdtempSync(path.join(os.tmpdir(), "fleet-regressions-")); + for (const f of mine) { + let result: Omit; + if (f.kind !== "maestro") { + result = { outcome: "skipped", detail: `a ${f.kind} flow; ui-test runs Maestro flows only` }; + } else { + try { + const text = prepareFlow(f.flow, { appId: o.appId, flowsDir: o.flowsDir }); + const pkg = flowAppId(text) ?? o.appId ?? ""; + const file = path.join(dir, `finding-${f.finding_id}.yaml`); + writeFileSync(file, text); + const before = pkg ? await deps.readCrashes(o.target, pkg) : null; + let flowError = await deps.runFlow(o.target, file, path.join(dir, `run-${f.finding_id}`), + { ...(o.appId ? { APP_ID: o.appId } : {}), ...(o.env ?? {}) }, o.timeoutMs ?? 300_000); + if (flowError && o.secrets?.length) flowError = redact(flowError, o.secrets); + const after = pkg ? await deps.readCrashes(o.target, pkg) : null; + result = judgeRegression({ check: f.check, platform: o.target.platform, flowError, before, after }); + } catch (e) { + result = { outcome: "failed", detail: `could not run the flow: ${(e as Error).message.slice(0, 300)}` }; + } + } + out.push({ finding_id: f.finding_id, ...result }); + await deps.report(f.finding_id, { + job_id: o.job.job_id, device_id: o.target.id, build, outcome: result.outcome, detail: result.detail, + }).catch(() => { /* the device row still carries it; a missed report is not a failed test */ }); + } + return out; +} diff --git a/docs/agents.md b/docs/agents.md index 5c703e8..acea22a 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -62,6 +62,9 @@ the file name to write. `--out -` writes to stdout. | `fleetr retry ID [--device ID] [--pool P] [--wait]` | Run again under a new id; `--wait` waits and exits with the verdict | | `fleetr results [JOB] [--device] [--workload] [--job] [--failures] [--limit N]` | Result rows, final rows only unless a job is named; default 20 | | `fleetr flaky [--device ID] [--app NAME]` | Builds that both passed and failed on one device | +| `fleetr findings [--app A] [--state open\|triaged\|all] [--verdict V] [--check C] [--since 24h] [--limit N] [--page N]` | What night QA found, newest sighting first; default 50 per page | +| `fleetr finding ID [--flow FILE\|-]` | One finding: steps, evidence, regression test, verdict history. `--flow` writes its Maestro (or tvloop) replay flow; `-` or no file is stdout | +| `fleetr verdict ID real\|duplicate\|not-a-bug\|agent-mistake\|fixed\|reopen [--note T] [--duplicate-of ID]` | Judge a finding. `fixed` only after `real`, once a fix has merged; exit 5 (`conflict`) otherwise | | `fleetr whoami` | Which collector, its auth mode, who you are to it and your scopes | | `fleetr login [--url U] [--token T]` · `fleetr logout [--url U]` | Save a token for a collector (from `--token` or stdin), or forget it | @@ -71,6 +74,59 @@ Which collector: `--url`, then `FLEET_RUNNER_URL`, then `FLEET_URL`, then overrides the `source` recorded on jobs (`cli` by default), for example `FLEET_SOURCE=ci` in a workflow. +## A morning brief + +`fleetr findings --since 24h --json` is the input for a scheduled agent +session that starts from last night's findings. This repository does not build +that session; it makes the input exist, in a shape that will not change +without a version note in the changelog: + +```json +{ + "items": [ + { + "id": 42, "app": "greenfolio", "build": "1.4 (12)", "platform": "android", + "device_id": "R58M...", "job_id": "explore-...", "mission_id": "add-a-plant", + "check": "crash", "severity": "high", + "title": "Crash: java.lang.NullPointerException", "detail": "...", + "screen": "02c1db49ee", "screen_name": "Add plant", + "steps": ["On \"My plants\":", " Tap \"Add plant\"", "..."], + "replay": { "kind": "maestro", "sha256": "...", "attempts": 2, "reproduced": 2 }, + "replay_label": "reproduced 2/2", + "artifacts": { "shot": "...", "sheet": "...", "trajectory": "...", "replay": "..." }, + "first_seen": "2026-10-15T02:14:09Z", "last_seen": "2026-10-15T02:14:09Z", + "seen_count": 1, "builds_seen": ["1.4 (12)"], + "verdict": null, "verdict_note": null, "verdict_at": null, "duplicate_of": null, + "status": "open", "issue": null, "regression": null + } + ], + "page": 1, "per_page": 50, "total": 1, "pages": 1, + "facets": { "apps": ["greenfolio"], "checks": ["crash"], "counts": { "open": 1, "triaged": 0, "total": 1 } } +} +``` + +The rules for reading it: + +- `since` filters on **last seen**, so a finding first seen a week ago that + came back last night is in the list; `first_seen` tells them apart. +- `verdict: null` is open: nobody has judged it, or a regression test + reopened it (`fleetr finding ID` shows a `regressed on build ...` line). +- `agent_mistake` and `not_a_bug` are not bugs in the app. +- `replay_label` is the evidence: `reproduced 2/2` is solid, `flaky 1/2` is + weaker, `crash log` means the platform's crash log is the evidence. +- `regression` is `null`, or the state of the test it became: `candidate` + (real), `active` (fixed, in the nightly) or `retired`. +- The list envelope is the control API's; `--page N` and `--limit N` page it, + and `total` says how many match. `facets` lists every app and check (not + only the filtered ones) and counts open and triaged findings for the app, + check and `since` asked for. The Finding schema in `openapi.json` is the + full contract. + +Then `fleetr finding ID --flow repro.yaml` gives the flow to reproduce it with +(`maestro test repro.yaml`, or `fleetr run ui-test` with it in a flows +directory), and once a fix has merged, `fleetr verdict ID fixed --note "PR +#123"` makes it a regression test in the app's nightly. + ## The skill ```bash @@ -81,7 +137,8 @@ fleet skill show A short file telling an agent the order to call things in (devices, workloads, validate, run, wait, artifacts), what each verdict and exit code -means, and what not to do: report a skip as a pass, pin someone else's device, +means, how to read a finding (`agent_mistake` is not a bug; never mark one +`fixed` without a fix merged), and what not to do: report a skip as a pass, pin someone else's device, retry a failure without reading it, poll in a loop, or run `shell`. Its tool and exit-code tables are generated from the code and checked by the test suite, so they cannot fall behind the binary. @@ -101,7 +158,10 @@ and `"--token", "fr_..."` if it has tokens on (or set `FLEET_RUNNER_TOKEN` in the server's `env`; a saved `fleetr login` for that URL also works). Tools: `fleet_devices`, `fleet_workloads`, `fleet_validate`, `fleet_run`, `fleet_job`, `fleet_cancel`, `fleet_retry`, `fleet_results`, `fleet_flaky`, -`fleet_artifact`, `fleet_whoami`. Each carries read-only and destructive +`fleet_artifact`, `fleet_findings`, `fleet_finding`, `fleet_verdict`, +`fleet_whoami`. `fleet_finding` returns the replay flow's text inline when it +is 16 KiB or less (nearly always); `fleet_verdict` is the one findings tool +that changes anything, and needs the `run` scope. Each carries read-only and destructive annotations and, where it returns a summary, an output schema. Screenshots come back as images. Jobs, workloads, devices and artifacts are also readable as resources (`fleet://job/{id}` and so on). diff --git a/docs/api.md b/docs/api.md index 1f8c7da..61494c5 100644 --- a/docs/api.md +++ b/docs/api.md @@ -25,14 +25,17 @@ disagree: the test suite fails if they do. | `GET /api/v1/workloads` · `/workloads/{name}` | read | What the fleet can run: executor, params schema, a valid example spec, how many live agents can take it | | `GET /api/v1/templates` | read | Saved job templates | | `GET /api/v1/jobs` | read | Jobs; filter by `status` (`waiting`, `queued`, `claimed`, `done`, `failed`, `cancelled`, `skipped`), `workload`, `executor` (`device` or `host`), `device`, `pool`, `source`, `requested_by`, `mine=true`, `q` (substring of the job id), `from`/`to`, `has_error=true`. The list carries `facets` over the whole table: `status_counts`, `workloads`, `pools` | -| `POST /api/v1/jobs` | run | Enqueue. 201 with the job id, its status, and links to its summary and wait URLs. 422 `no_capable_agent` when no live agent declares the workload (or backend); 400 `no_matching_devices` for a fan-out that matches nothing, and 400 `no_build` when `latest` names no published build | +| `POST /api/v1/jobs` | run | Enqueue. 201 with the job id, its status, and links to its summary and wait URLs. 422 `no_capable_agent` when no live agent declares the workload (or backend), or `unresolved_reference`; 400 `no_matching_devices` when the targets match no registered device, `no_build` when `latest` names no published build, and `invalid_spec`, `invalid_match`, `invalid_dependency`, `secret_in_spec`. The OpenAPI document lists the same | | `POST /api/v1/jobs/validate` | run | Dry run: schema errors by path, the devices it would land on, warnings. Always 200 | | `GET /api/v1/jobs/{id}` | read | One job, with its spec | | `GET /api/v1/jobs/{id}/summary` | read | The verdict across every device, per-device outcomes and artifact refs. Works for a fan-out parent | | `GET /api/v1/jobs/{id}/wait?timeout_s=` | read | Long-poll (≤ 120 s) until the job finishes; returns the summary | | `POST /api/v1/jobs/{id}/cancel` · `/retry` | run | Cancel (cascades over a fan-out) or run again under a new id | | `GET /api/v1/results` · `/results/flaky` | read | Result rows (`include=payload` for the raw row), and tests whose verdict flips on one build | -| `GET /api/v1/artifacts/{sha256}` · `/content` | read | What an artifact is, and its bytes. Active types (HTML, SVG) are served as attachments with a sandbox CSP | +| `GET /api/v1/artifacts/{sha256}` · `/content` | read | What an artifact is, and its bytes. `channel` is `pr` for a pull request's build (`latest` never resolves to it) and `main` otherwise. Active types (HTML, SVG) are served as attachments with a sandbox CSP | +| `GET /api/v1/findings` | read | What night QA found; filter by `app`, `state` (`open`, `triaged`, `all`), `verdict` (comma-separated), `check`, `since` (ISO 8601 or `24h`, `7d`). See [Findings](#findings) | +| `GET /api/v1/findings/{id}` | read | One finding: steps in words, evidence as links, the replay flow (inline up to 16 KiB), its regression test, verdict history, counts | +| `POST /api/v1/findings/{id}/verdict` | run | `{verdict, note?, duplicate_of?}`: `real`, `duplicate`, `not_a_bug`, `agent_mistake`, `fixed` (only after `real`; 409 `conflict` otherwise) or `null` to reopen. Audited | | `GET /api/v1/schedules` · `/alerts` | read | Recurring jobs and open alerts | | `GET /api/v1/audit` | admin | Who did what, when tokens are on | | `GET /metrics` | read | The fleet in Prometheus text format: devices by status, jobs by status, oldest queued job, per-device success ratio over 24 h, open alerts, artifact counts. Computed from the database on each scrape. Not under `/api/v1` and not in the OpenAPI document | @@ -74,11 +77,13 @@ unverified; with tokens on, `requested_by` is the token's name, or the person a **The older routes are deprecated, not removed.** `GET /api/devices`, `GET /api/devices/{id}`, `GET /api/jobs`, `GET /api/jobs/{id}`, -`GET /api/results`, `GET /devices`, `GET /schedules`, `GET /jobs/{id}` and -`POST /jobs` · `/api/jobs` answer with `Deprecation` (RFC 9745) and a -`Link: <...>; rel="successor-version"` header. Only those exact routes carry -the headers: `/api/devices/{id}/beacons`, `/api/results/bench` and the rest of -the dashboard's read API do not. There is no sunset date. The runner protocol is +`GET /api/results`, `GET /devices`, `GET /schedules`, `GET /jobs/{id}`, +`POST /jobs` · `/api/jobs`, and `GET /api/findings`, `GET /api/findings/{id}` +and `POST /api/findings/{id}/verdict` answer with `Deprecation` (RFC 9745) and +a `Link: <...>; rel="successor-version"` header. Only those exact routes carry +the headers: `/api/devices/{id}/beacons`, `/api/results/bench`, +`/api/findings/precision`, `/digest`, `/issues` and the rest of the +dashboard's read API do not. There is no sunset date. The runner protocol is not deprecated. Timestamps are ISO-8601 **UTC with a `Z`**. SQLite stores @@ -104,6 +109,8 @@ artifacts). See [the protocol](protocol.md#authentication-optional) and | `POST /artifacts` | Upload bytes; returns `sha256`. Headers: `x-artifact-name`, and for a build `x-artifact-app`, `x-artifact-build`, `x-artifact-platform`. `x-artifact-channel: pr` marks a pull request's build, which `"sha256": "latest"` never resolves to; any other channel value is a 400. See [Publish builds on merge](integration/publish-on-merge.md#pull-request-builds-never-move-latest) | | `GET /artifacts/:sha256` | Download, supports Range | | `POST /findings` | An `explore` finding from an executor; merged on its fingerprint. See [Findings and verdicts](explore/findings.md#the-api) | +| `GET /findings/regressions?app=&state=active&platform=` | A nightly `ui-test` pulling an app's regression flows. See [Regression tests](explore/findings.md#regression-tests) | +| `POST /findings/:id/regression` | An executor reporting one regression run: `{job_id, device_id, build, outcome, detail}`; `reproduced` reopens the finding | | `POST /api/jobs/:id/mirror` | One raw JPEG frame from an executor driving a device, for the live view. Kept in memory only (the last three frames), never stored; 2 MB cap per frame. The one row here that `FLEET_DASH_TOKEN` does guard; the `explore` workload sends it | ## Queue @@ -183,12 +190,17 @@ From the [`explore`](explore/index.md) workload. Full shapes and replies are in | Method & path | Purpose | |---|---| -| `GET /api/findings` | The list: `?app= &check= &status=open\|triaged\|all &since= &limit=` | -| `GET /api/findings/:id` | One finding, with its duplicates | +| `GET /api/findings` | The list: `?app= &check= &status=open\|triaged\|all &verdict= &since= &limit= &offset=` | +| `GET /api/findings/:id` | One finding, with its duplicates, evidence links, replay flow, regression test and history | | `GET /api/findings/precision` | How often each check and visual class was judged real; which are switched off | | `GET /api/findings/digest?hours=` | The morning digest, as text and as data | | `GET /api/findings/issues` | Every GitHub issue composed, dry run or filed | -| `POST /api/findings/:id/verdict` | Real, duplicate, not a bug, agent's mistake, or reopen. Guarded | +| `POST /api/findings/:id/verdict` | Real, duplicate, not a bug, agent's mistake, fixed (only after real), or reopen. Guarded | + +The control API has the list, the detail and the verdict too +(`/api/v1/findings`, above). Both call the same functions in +`collector/src/findings.ts`; the v1 verdict needs the `run` scope with tokens +on, and the dashboard's route here keeps the `admin` it always needed. ## Mutations diff --git a/docs/dashboard.md b/docs/dashboard.md index ceaee02..ddabd09 100644 --- a/docs/dashboard.md +++ b/docs/dashboard.md @@ -64,10 +64,17 @@ not allow, is refused with each problem named by path, where the older `POST /api/jobs` took the spec and let the job fail later. The curl it shows is the same request. Jobs it enqueues record `source: dash`. +**Findings read and judge through `/api/v1/findings`**, so a verdict from the +dashboard is the same call, with the same refusals, as `fleetr verdict` and +`fleet_verdict`. The precision table still reads `/api/findings/precision`, +which has no `/api/v1` counterpart. + Alerts appear as a banner on every screen. The layout works on a phone, and `?` lists the keyboard shortcuts — `g j` jobs, `g d` devices, `g f` findings, `g n` new job, -`/` search, and on a finding `r`, `d`, `n` or `a` for its verdict (real, -duplicate, not a bug, the agent's mistake). +`/` search, and on a finding `r`, `d`, `n`, `a` or `f` for its verdict (real, +duplicate, not a bug, the agent's mistake, fixed). Fixed is offered only once a +finding is real, and a finding with a regression test shows its state, last +run and verdict history in a panel of its own. ## When the collector wants a token diff --git a/docs/explore/findings.md b/docs/explore/findings.md index 6e9b3a5..5b2fd38 100644 --- a/docs/explore/findings.md +++ b/docs/explore/findings.md @@ -15,7 +15,7 @@ builds it has been seen, how it replayed ("reproduced 2/2", "flaky 1/2", The **detail** view shows the screenshot, the steps in words, the detail text with every replay's outcome, links to the contact sheet, trajectory, log and -replay file, and four buttons: +replay file, and five buttons: | Button | Key | Means | Counts towards precision | |---|---|---|---| @@ -23,12 +23,19 @@ replay file, and four buttons: | **Duplicate** | d | The same problem as another finding (pick which) | No | | **Not a bug** | n | The app is fine; the check was wrong | Yes, as a miss | | **Agent's mistake** | a | The explorer did something odd and reported its own doing | Yes, as a miss | +| **Fixed** | f | Real, and a fix has merged. Only after Real | Yes, as a hit | A note is optional and is kept with the verdict. Setting the verdict to none reopens a finding. A duplicate follows its chain to the original, and a loop is refused. The keys are ignored straight after g, so g r still goes to Results. +**Fixed** is greyed out until a finding is Real (the collector refuses it +otherwise, with a 409). It turns the finding's replay flow into a regression +test; see [Regression tests](#regression-tests). Once a finding has one, a +**Regression test** panel shows its state, its last run and the verdict +history. + A **precision table** at the top shows, for every check that has verdicts, how often it was right and whether it has been switched off. @@ -49,6 +56,58 @@ on its own; the judge is told not to report it and the rest keep running. Any other check is switched off whole. **Crashes and ANRs are never switched off**: their evidence is the platform's own log, not a judgement. +## Regression tests + +A finding that was real and is fixed becomes a test, with no hand-copying: + +| Verdict | What happens to the finding's replay flow | +|---|---| +| **Real** | Copied into the collector's `regressions` table as a **candidate**. Not run yet | +| **Fixed** (only after Real) | The candidate becomes **active**. A finding marked Real before this release goes straight to active | +| **Duplicate**, **Not a bug**, **Agent's mistake** | A candidate or active test is **retired** | +| Real again, on a retired one | Back to **candidate** | +| Reopen (none) | Left as it is | + +The flow is copied into the database, not written to the lab's flows +directory: that directory is outside git and outside any backup this +repository controls, and a copy in the database also outlives the artifact +store's garbage collection. A finding with no replay file (some crashes) or +whose file the store has already collected gets no test, and the verdict's +answer says so. + +**How a test runs.** Nothing is scheduled automatically. An app's nightly +`ui-test` opts in once with `params.regressions` (the app key the findings are +filed under, or `true` for `app.name`), and from then on every finding marked +Fixed is in the next night's run. The executor pulls the app's active flows +from the collector at run time and runs each, after the suite's own flow, on +every device of the same platform. See [ui-test](../workloads/host.md#ui-test). + +**What it asserts.** For a **crash** or **ANR** finding on Android, the +executor reads the crash buffer (`logcat -b crash`, and `am_anr` in the events +buffer) before and after the flow; a new crash of the package during it is the +bug coming back. For **every other check** (frozen, blank, left_app, a11y, +visual, goal, dead_control) the oracle that found the bug was a model or a +heuristic the flow does not carry, so the honest claim is weaker: **the flow +replays without error**. Such a flow failing fails the device's row but does +not reopen anything, because a step that no longer replays may be the bug or +may be a redesigned screen. On iOS no crash buffer is read, so a crash +finding's flow is also "replays without error" there, and its detail says so. +tvloop flows (a Roku's) are kept but not run: ui-test drives Maestro. + +**When it comes back.** A run that reproduces the crash reopens the finding: +the verdict is cleared, it is counted as seen once more, the build is added to +its builds, and its history gets a line such as `regressed on build 1.4 (12) +(job nightly-ui-…, R58M…): the app crashed again during the flow: …`. The +device's row fails with `regression #N reproduced (finding reopened)`. Three +devices reproducing it in one job reopen it once. The test stays active, so +the nightly keeps failing until the finding is marked Fixed again. + +**Not verified:** no nightly on the lab has run a regression flow yet; the +logic is covered by the collector's suite against fake crash readings and a +fake Maestro, not by a phone. GitHub issues are only ever created, never read +back, so an issue being closed does not mark a finding fixed; the verdict is +the only trigger. + ## The morning digest Set on the collector: @@ -152,8 +211,8 @@ finding. | Method & path | | |---|---| -| `GET /api/findings?app=&check=&status=open\|triaged\|all&since=&limit=` | Newest last-seen first; up to 500 | -| `GET /api/findings/:id` | One finding, with its duplicates | +| `GET /api/findings?app=&check=&status=open\|triaged\|all&verdict=&since=&limit=&offset=` | Newest last-seen first; up to 500. `since` takes ISO 8601 or a span such as `24h` | +| `GET /api/findings/:id` | One finding, with its duplicates, evidence links, replay flow, regression test and verdict history | | `GET /api/findings/precision?app=&min=10` | `{floor, min, classes[], disabled[]}`; the workload reads `disabled` at the start of a night | | `GET /api/findings/digest?hours=12` | The digest, as text and as data | | `GET /api/findings/issues` | Every issue composed, dry run or filed | @@ -162,7 +221,24 @@ finding. | Method & path | | |---|---| -| `POST /api/findings/:id/verdict` | `{verdict: "real"\|"duplicate"\|"not_a_bug"\|"agent_mistake"\|null, note?, duplicate_of?}`. Needs `X-Fleet-Token` when `FLEET_DASH_TOKEN` is set | +| `POST /api/findings/:id/verdict` | `{verdict: "real"\|"duplicate"\|"not_a_bug"\|"agent_mistake"\|"fixed"\|null, note?, duplicate_of?}`. Needs `X-Fleet-Token` when `FLEET_DASH_TOKEN` is set. `fixed` before `real` is a 409. Answers with the finding, its regression test and `change`, one sentence on what the verdict did to the test | + +The same reads and the verdict are in the control API as `GET +/api/v1/findings`, `GET /api/v1/findings/{id}` and `POST +/api/v1/findings/{id}/verdict` (scope `run`, audited); both call the same +functions, and the dashboard uses the v1 ones. The older list, detail and +verdict routes still answer, with `Deprecation` headers naming their +successors; precision, digest and issues are only here. See [Control +API](../api.md#findings). + +### Regression runs (executors) + +Protocol routes, like `POST /findings`: + +| Method & path | | +|---|---| +| `GET /findings/regressions?app=&state=active&platform=` | `{regressions: [...]}`, each with its flow text. `state` is `candidate`, `active` (default), `retired` or `all` | +| `POST /findings/:id/regression` | `{job_id, device_id, build, outcome: "passed"\|"reproduced"\|"failed"\|"skipped", detail}`. `reproduced` on an active test reopens the finding | ## Retention diff --git a/docs/workloads/host.md b/docs/workloads/host.md index 499df54..17a67e9 100644 --- a/docs/workloads/host.md +++ b/docs/workloads/host.md @@ -279,6 +279,50 @@ all, the error is its exit code and the last line it printed to stderr, never its command line, which carries the test account's password. Until 0.9 the reason was in the uploaded report and nowhere else. +### Regression tests from fixed findings + +`params.regressions` adds the app's fixed night-QA findings to the run. Set it +to the app key the findings are filed under (what `fleetr findings` shows in +the APP column), or `true` to use `app.name`: + +```json +{ "workload": "ui-test", "executor": "host", + "app": { "name": "greenfolio-android", "build": "nightly", "sha256": "latest" }, + "suite": { "kind": "maestro", "flows": "greenfolio/smoke.yaml", "app_id": "com.taylab.greenfolio.smoke" }, + "params": { "regressions": "greenfolio" }, + "targets": { "match": "os ~ 'android'", "exclusive": true } } +``` + +At run time the executor asks the collector for the app's **active** +regression flows (`GET /findings/regressions`), and after the suite's own flow +runs each one whose finding was on the same platform as the device. A finding +joins by being marked Real and then Fixed; nobody edits the schedule. `flows` +may be left out to run only the regression flows; a device with none to run is +then skipped, not passed. + +Each stored flow is adapted before it runs: its `appId` becomes the job's +`suite.app_id` when the job sets one (night QA often explores `.debug`; the +nightly may test `.smoke`), and a relative `runFlow` (the mission's sign-in +flow) is resolved under the flows directory. The suite's sign-in variables +reach the flow as they reach the suite's. + +What a run asserts depends on the finding's check: + +| Finding | On Android | Elsewhere | +|---|---|---| +| `crash`, `anr` | No new crash or ANR of the package in `logcat -b crash` / `am_anr` during the flow. A new one is `reproduced` and **reopens the finding** | Replays without error; the crash is not checked, and the detail says so | +| Any other check | Replays without error | Replays without error | + +"Replays without error" is all a replay can claim for a check whose oracle was +a model or a heuristic: the flow repeats the steps, not the judgement. A flow +that fails is `failed` and fails the row, without reopening the finding, +because a step that no longer replays may be a redesigned screen. tvloop flows +are skipped (ui-test drives Maestro). Each regression counts as one test in +the row's `passed`/`failed`, is listed in `test.regressions` with its outcome, +and a failing one is named in the row's `error`. The regressions run outside +`params.network` shaping. Not yet run on the lab; see [Regression +tests](../explore/findings.md#regression-tests). + **iOS UI tests need a Mac with full Xcode.** `simctl` and `devicectl` ship with Xcode, not the Command Line Tools, so they run on a different executor from the Android work. [The iOS executor](../deploy/ios-executor.md) covers standing one diff --git a/fleet/src/mcp/protocol.ts b/fleet/src/mcp/protocol.ts index 456a670..81c9051 100644 --- a/fleet/src/mcp/protocol.ts +++ b/fleet/src/mcp/protocol.ts @@ -32,7 +32,9 @@ const INSTRUCTIONS = (where: string | null) => "The usual order: fleet_devices (what is online) -> fleet_workloads name= (a valid example spec) -> " + "fleet_validate (fix what it says) -> fleet_run (waits for the verdict) -> fleet_artifact (screenshots, logs).\n" + "A verdict of skipped means nothing ran; never report it as a pass. Do not retry a failed job more than once " + - "without reading its error. Prefer targets.pool or targets.match over pinning a device_id another person may be using." + + "without reading its error. Prefer targets.pool or targets.match over pinning a device_id another person may be using.\n" + + "Night QA's findings: fleet_findings since=24h (last night) -> fleet_finding id= (steps and the replay flow). " + + "agent_mistake is not a bug; mark a finding fixed with fleet_verdict only after a fix has merged." + (where ? `\n\nCollector: ${where}` : ""); /** One message in, zero or one out. */ diff --git a/fleet/src/mcp/tools.ts b/fleet/src/mcp/tools.ts index 59d95c3..b8388bb 100644 --- a/fleet/src/mcp/tools.ts +++ b/fleet/src/mcp/tools.ts @@ -71,6 +71,7 @@ const NEXT: Record = { unauthorized: "The gateway or CLI has no valid token for the collector.", unreachable: "The collector did not answer. It may be down or the URL may be wrong.", rate_limited: "Too many calls; wait a minute.", + conflict: "The state does not allow it. For fleet_verdict fixed: mark the finding real first, and fixed once the fix has merged.", }; export function refusal(name: string, e: unknown, remote: boolean): ToolResult { @@ -197,6 +198,60 @@ const SUMMARY_OUT: Record = { const READ = { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false }; +const FINDING_ITEM: Record = { + type: "object", + required: ["id", "app", "check", "severity", "title", "verdict", "replay", "seen_count"], + properties: { + id: { type: "integer" }, app: { type: "string" }, check: { type: "string" }, severity: { type: "string" }, + title: { type: "string" }, screen: { type: "string" }, platform: { type: "string" }, + verdict: { type: ["string", "null"], description: "null: open. agent_mistake: not a bug in the app." }, + replay: { type: "string", description: "reproduced 2/2, flaky 1/2, crash log." }, + seen_count: { type: "integer" }, builds: { type: "integer" }, last_seen: { type: ["string", "null"] }, + regression: { type: ["string", "null"], description: "candidate, active (in the nightly) or retired." }, + }, +}; + +const FINDINGS_OUT: Record = { + type: "object", + required: ["findings", "total"], + properties: { findings: { type: "array", items: FINDING_ITEM }, total: { type: "integer" } }, +}; + +const FINDING_OUT: Record = { + type: "object", + required: ["id", "app", "check", "title", "verdict", "steps", "evidence", "history"], + properties: { + ...(FINDING_ITEM.properties as Record), + detail: { type: "string" }, + steps: { type: "array", items: { type: "string" } }, + evidence: { type: "array", items: { type: "object" } }, + replay_flow: { type: ["object", "null"], description: "kind, sha256, and text when the flow is small enough to inline." }, + regression_test: { type: ["object", "null"] }, + history: { type: "array", items: { type: "object" } }, + note: { type: "string" }, + }, +}; + +const VERDICT_OUT: Record = { + type: "object", + required: ["id", "verdict"], + properties: { id: { type: "integer" }, verdict: { type: ["string", "null"] }, change: { type: ["string", "null"] }, regression: { type: ["string", "null"] } }, +}; + +type FindingRow = { + id: number; app: string; check: string; severity: string; title: string; screen_name: string; platform: string; + verdict: string | null; replay_label: string; seen_count: number; builds_seen: string[]; last_seen: string | null; + regression: { state: string } | null; +}; + +const findingItem = (f: FindingRow) => ({ + id: f.id, app: f.app, check: f.check, severity: f.severity, title: f.title, screen: f.screen_name, platform: f.platform, + verdict: f.verdict, replay: f.replay_label, seen_count: f.seen_count, builds: f.builds_seen.length, last_seen: f.last_seen, + regression: f.regression?.state ?? null, +}); + +const VERDICT_WORDS = ["real", "duplicate", "not_a_bug", "agent_mistake", "fixed"] as const; + // --- the tools --------------------------------------------------------------------------- export const TOOLS: Tool[] = [ @@ -466,6 +521,119 @@ export const TOOLS: Tool[] = [ return readArtifact(client, sha, Math.max(500, Math.min(Number(args.max_chars ?? 20_000) || 20_000, 200_000))); }, }, + { + name: "fleet_findings", + title: "What night QA found", + cli: "fleetr findings [--app A] [--since 24h]", + description: + "Bugs night QA (the explore workload) found on real devices, newest sighting first. Each was replayed on a clean " + + "install before it was filed: `replay` says how often it reproduced (\"flaky 1/2\" is weaker evidence). verdict " + + "null is open; agent_mistake means the model misread the app and is NOT a bug. since=\"24h\" is last night. " + + "fleet_finding with an id gives the steps, evidence and the replay flow.", + inputSchema: { + type: "object", + properties: { + app: { type: "string", description: "The explore app key, e.g. greenfolio." }, + state: { type: "string", enum: ["open", "triaged", "all"], description: "Default all." }, + verdict: { type: "string", description: "Comma-separated: real, duplicate, not_a_bug, agent_mistake, fixed." }, + check: { type: "string", description: "crash, anr, frozen, blank, left_app, a11y, visual, goal, dead_control." }, + since: { type: "string", description: "ISO 8601, or a span back from now: 24h, 7d." }, + limit: { type: "number", description: "Default 30, at most 200." }, + }, + }, + outputSchema: FINDINGS_OUT, + annotations: READ, + needs: "read", + async run({ client }, args) { + const s = (k: string) => (typeof args[k] === "string" && args[k] ? (args[k] as string) : undefined); + const page = await client.listFindings({ + app: s("app"), state: s("state") as "open" | "triaged" | "all" | undefined, verdict: s("verdict"), check: s("check"), since: s("since"), + per_page: Math.max(1, Math.min(Number(args.limit ?? 30) || 30, 200)), + }); + const out = { findings: page.items.map((f) => findingItem(f as FindingRow)), total: page.total }; + return ok(out.findings.length ? out : "No findings match.", out); + }, + }, + { + name: "fleet_finding", + title: "One finding, with its replay flow", + cli: "fleetr finding ID [--flow FILE]", + description: + "One finding by id: what went wrong, the steps in words, links to the screenshot, contact sheet, trajectory and " + + "log (open them with fleet_artifact), its verdict history, its regression test, and the Maestro (or tvloop) flow " + + "that replays it -- inline when small, so you can reproduce the bug with fleet_run or locally.", + inputSchema: { + type: "object", + required: ["id"], + properties: { + id: { type: "integer", description: "The finding id." }, + include_flow: { type: "boolean", description: "Include the replay flow text when it is small. Default true." }, + }, + }, + outputSchema: FINDING_OUT, + annotations: READ, + needs: "read", + async run({ client }, args) { + const id = Number(args.id); + if (!Number.isInteger(id) || id < 1) return failure("id must be a finding id (a positive integer)"); + const f = await client.getFinding(String(id)); + const flow = f.replay_flow + ? { kind: f.replay_flow.kind, sha256: f.replay_flow.sha256, ...(args.include_flow !== false && f.replay_flow.text !== null ? { text: f.replay_flow.text } : {}), + ...(f.replay_flow.truncated ? { note: "too big to inline; fleet_artifact with this sha256 returns it" } : {}) } + : null; + const out = { + ...findingItem(f as FindingRow), + detail: clip(f.detail, 4000) ?? "", + steps: f.steps, + evidence: f.evidence.map((e) => ({ role: e.role, sha256: e.sha256, in_store: e.in_store })), + replay_flow: flow, + regression_test: f.regression + ? { state: f.regression.state, asserts: f.regression.asserts, runs: f.regression.runs, last_outcome: f.regression.last_outcome ?? null, last_detail: f.regression.last_detail ?? null } + : null, + history: f.history, + ...(f.verdict === "agent_mistake" ? { note: "agent_mistake: the model misread the app. Not a bug; do not fix the app for it." } : {}), + }; + const result = ok(out, out); + if (f.artifacts.shot) result.content.push({ type: "resource_link", uri: `fleet://artifact/${f.artifacts.shot}`, name: "screenshot", mimeType: "image/png" }); + return result; + }, + }, + { + name: "fleet_verdict", + title: "Judge a finding", + cli: "fleetr verdict ID VERDICT [--note TEXT]", + description: + "Record a verdict on a finding. real: a bug in the app (its replay flow is kept as a candidate regression test). " + + "fixed: only after real, and only once a fix has MERGED -- the flow then runs in the app's nightly and reopens the " + + "finding if the crash comes back. not_a_bug / agent_mistake: the check or the model was wrong (retires any test). " + + "duplicate needs duplicate_of. Verdicts decide which checks night QA keeps running, so do not guess.", + inputSchema: { + type: "object", + required: ["id", "verdict"], + properties: { + id: { type: "integer" }, + verdict: { type: "string", enum: [...VERDICT_WORDS] }, + note: { type: "string", description: "Why, for whoever reads it next. Name the fix's PR when marking fixed." }, + duplicate_of: { type: "integer", description: "With duplicate: the finding this one duplicates." }, + }, + }, + outputSchema: VERDICT_OUT, + annotations: { readOnlyHint: false, destructiveHint: false, idempotentHint: true, openWorldHint: false }, + needs: "run", + async run({ client }, args) { + const id = Number(args.id); + if (!Number.isInteger(id) || id < 1) return failure("id must be a finding id (a positive integer)"); + const verdict = String(args.verdict ?? "").replace(/-/g, "_"); + if (!(VERDICT_WORDS as readonly string[]).includes(verdict)) return failure(`verdict must be one of ${VERDICT_WORDS.join(", ")}`); + const r = await client.setFindingVerdict(String(id), { + verdict: verdict as (typeof VERDICT_WORDS)[number], + ...(typeof args.note === "string" && args.note ? { note: args.note } : {}), + ...(args.duplicate_of !== undefined ? { duplicate_of: Number(args.duplicate_of) } : {}), + }); + const out = { id, verdict: r.finding.verdict ?? null, change: r.change ?? null, regression: r.finding.regression?.state ?? null }; + return ok(out, out); + }, + }, { name: "fleet_whoami", title: "Who am I to the fleet", diff --git a/fleet/src/work.ts b/fleet/src/work.ts index 128cc5c..39a1572 100644 --- a/fleet/src/work.ts +++ b/fleet/src/work.ts @@ -17,7 +17,7 @@ import { homedir } from "node:os"; import path from "node:path"; import { Readable } from "node:stream"; import { pipeline } from "node:stream/promises"; -import type { FleetClient, JobSpecInput, JobSummary } from "../../collector/src/client/index.js"; +import type { FindingVerdictInput, FleetClient, JobSpecInput, JobSummary } from "../../collector/src/client/index.js"; import { FleetApiError } from "../../collector/src/client/index.js"; import { childEnv, load } from "./config.js"; import { connect, readCredentials, resolveBrain, writeCredentials } from "./connect.js"; @@ -326,6 +326,109 @@ async function artifact(client: FleetClient, a: Args) { return EXIT.passed; } +/** `not-a-bug` and `agent-mistake` as typed, the API's names, or `reopen` for null. */ +export function verdictWord(raw: string | undefined): FindingVerdictInput["verdict"] | undefined { + if (!raw) return undefined; + const w = raw.trim().toLowerCase().replace(/-/g, "_"); + if (w === "reopen" || w === "open") return null; + return (["real", "duplicate", "not_a_bug", "agent_mistake", "fixed"] as const).find((v) => v === w); +} + +async function findings(client: FleetClient, a: Args) { + const page = await client.listFindings({ + app: str(a, "app"), state: str(a, "state") as "open" | "triaged" | "all" | undefined, verdict: str(a, "verdict"), + check: str(a, "check"), since: str(a, "since"), per_page: num(a, "limit", 50), page: num(a, "page", 1), + }); + if (a.flags.json) return printJson(page), EXIT.passed; + if (!page.items.length) { + err("no findings match"); + return EXIT.passed; + } + table( + page.items.map((f) => [ + `#${f.id}`, f.app, f.check, f.severity, f.verdict ?? "open", f.replay_label, `${f.seen_count}x`, ago(f.last_seen), + f.regression ? f.regression.state : "", f.title.slice(0, 60), + ]), + ["ID", "APP", "CHECK", "SEV", "VERDICT", "REPLAY", "SEEN", "LAST", "REGRESSION", "TITLE"], + ); + if (page.total > page.items.length) err(`${page.items.length} of ${page.total}; --limit N or --page N for more`); + return EXIT.passed; +} + +function findingIdOf(a: Args, usage: string): number { + const raw = (a.positional[0] ?? "").replace(/^#/, ""); + if (!/^\d+$/.test(raw)) throw new UsageError(usage); + return Number(raw); +} + +async function finding(client: FleetClient, a: Args) { + const id = findingIdOf(a, "usage: fleet finding [--flow FILE|-] [--json]"); + const f = await client.getFinding(String(id)); + // --flow with no file is stdout, so `fleet finding 12 --flow > repro.yaml` works. + const dest = a.flags.flow === true ? "-" : str(a, "flow"); + if (dest) { + const rf = f.replay_flow; + if (!rf) { + err(`finding ${id} has no replay flow (${f.replay_label})`); + return EXIT.error; + } + let text = rf.text; + if (text === null) { + if (!rf.in_store) { + err(`finding ${id}'s replay flow ${rf.sha256} is no longer in the artifact store`); + return EXIT.error; + } + text = await (await client.getArtifactContent(rf.sha256, undefined, { timeoutMs: 60_000 })).text(); + } + if (dest === "-") process.stdout.write(text); + else { + writeFileSync(dest, text); + err(`wrote ${dest} (${rf.kind} flow, ${Buffer.byteLength(text)} bytes)`); + } + if (dest === "-" || a.flags.json !== true) return EXIT.passed; + } + if (a.flags.json) return printJson(f), EXIT.passed; + out(`#${f.id} ${f.title}`); + out(` ${f.app} · ${f.check} (${f.severity}) · ${f.verdict ?? "open"} · ${f.replay_label} · seen ${f.seen_count}x on ${f.builds_seen.length} build${f.builds_seen.length === 1 ? "" : "s"}`); + out(` ${f.screen_name || "unnamed screen"} on ${f.device_id} (${f.platform}), mission ${f.mission_id}, first found by ${f.job_id}`); + if (f.verdict === "agent_mistake") out(" agent_mistake: the model misread the app; this is not a bug in it."); + if (f.detail) out(`\n${f.detail}`); + if (f.steps.length) { + out("\nsteps"); + f.steps.forEach((s, i) => out(` ${i + 1}. ${s}`)); + } + if (f.evidence.length) { + out("\nevidence"); + for (const e of f.evidence) out(` ${e.label.padEnd(14)} ${e.sha256}${e.in_store ? "" : " (collected)"}`); + } + if (f.replay_flow) out(`\nreplay flow: ${f.replay_flow.kind}, ${f.replay_flow.sha256} (fleet finding ${f.id} --flow FILE writes it)`); + if (f.regression) { + const r = f.regression; + out(`regression test: ${r.state}, ${r.asserts === "no_crash" ? "asserts no crash" : "replays without error"}, run ${r.runs}x${r.last_outcome ? `, last ${r.last_outcome}` : ""}`); + } + if (f.history.length) { + out("\nhistory"); + for (const h of f.history) out(` ${h.at ?? ""} ${h.event === "regressed" ? "regressed" : h.verdict ?? "reopened"}${h.actor ? ` (${h.actor})` : ""}${h.note ? `: ${h.note}` : ""}`); + } + return EXIT.passed; +} + +async function verdict(client: FleetClient, a: Args) { + const usage = "usage: fleet verdict real|duplicate|not-a-bug|agent-mistake|fixed|reopen [--note TEXT] [--duplicate-of ID]"; + const id = findingIdOf(a, usage); + const v = verdictWord(a.positional[1]); + if (v === undefined) throw new UsageError(usage); + const dup = str(a, "duplicate-of"); + const r = await client.setFindingVerdict(String(id), { + verdict: v, + ...(str(a, "note") ? { note: str(a, "note") } : {}), + ...(dup ? { duplicate_of: Number(dup.replace(/^#/, "")) } : {}), + }); + if (a.flags.json) return printJson(r), EXIT.passed; + out(`#${id} ${r.finding.verdict ?? "reopened"}${r.change ? `; ${r.change}` : ""}`); + return EXIT.passed; +} + async function whoami(client: FleetClient, a: Args) { const [me, health] = await Promise.all([client.whoami(), client.getHealth()]); if (a.flags.json) return printJson({ ...me, url: client.baseUrl, version: health.version }), EXIT.passed; @@ -468,6 +571,11 @@ export const WORK_USAGE = ` fleet results [--device ID] [--workload W] [--job ID] [--failures] [--limit N] fleet flaky [--device ID] [--app NAME] UI tests that pass and fail on the same build. fleet artifact SHA [--out FILE|-] Describe or download an artifact. + fleet findings [--app A] [--state open|triaged|all] [--verdict V] [--check C] [--since 24h] [--limit N] + What night QA found. --since 24h --json is the morning brief. + fleet finding ID [--flow FILE|-] One finding: steps, evidence, history; --flow writes its replay flow. + fleet verdict ID real|duplicate|not-a-bug|agent-mistake|fixed|reopen [--note TEXT] [--duplicate-of ID] + fixed only after real, once the fix has merged: its flow joins the nightly. fleet login [--url U] [--token T] | fleet logout | fleet whoami fleet token create NAME --scope read,run,admin,agent,delegate | list | revoke ID|NAME fleet skill show | fleet skill install [--target cursor,claude,codex|DIR] @@ -476,7 +584,7 @@ export const WORK_USAGE = ` Exit codes: 0 passed, 1 failed, 2 usage/invalid spec, 3 still running, 4 skipped (nothing ran), 5 refused/unreachable. `; -export const WORK_COMMANDS = new Set(["devices", "workloads", "run", "job", "wait", "cancel", "retry", "results", "flaky", "artifact", "login", "logout", "whoami", "token", "skill"]); +export const WORK_COMMANDS = new Set(["devices", "workloads", "run", "job", "wait", "cancel", "retry", "results", "flaky", "artifact", "findings", "finding", "verdict", "login", "logout", "whoami", "token", "skill"]); export async function workCommand(command: string, argv: string[]): Promise { const a = parseArgs(argv); @@ -495,6 +603,9 @@ export async function workCommand(command: string, argv: string[]): Promise [--reason R]"); diff --git a/fleet/test/cli.test.ts b/fleet/test/cli.test.ts index 8d637b1..d74077a 100644 --- a/fleet/test/cli.test.ts +++ b/fleet/test/cli.test.ts @@ -7,7 +7,7 @@ import assert from "node:assert/strict"; import { spawn } from "node:child_process"; import { existsSync, mkdirSync, readFileSync, statSync, writeFileSync } from "node:fs"; import path from "node:path"; -import { fakeDevice, FLEET, startCollector, tempDir, TSX, type Collector } from "./helpers.js"; +import { fakeDevice, FLEET, postFinding, startCollector, tempDir, TSX, type Collector } from "./helpers.js"; let c: Collector; let device: Awaited>; @@ -160,3 +160,38 @@ test("skill show prints the skill; install writes it and will not clobber a stra assert.match(refused.stderr, /not overwriting/); assert.equal(readFileSync(file, "utf8"), "my own notes\n"); }); + +test("findings, finding --flow and verdict: the morning brief and a finding becoming a test", async () => { + const { id, flow } = await postFinding(c.base, "cliapp"); + const brief = await fleet(["findings", "--since", "24h", "--app", "cliapp", "--json"]); + assert.equal(brief.code, 0, brief.stderr); + const page = json(brief); + assert.deepEqual(Object.keys(page).sort(), ["facets", "items", "page", "pages", "per_page", "total"]); + assert.ok(page.facets.apps.includes("cliapp")); + assert.equal(page.items[0].id, id); + assert.equal(page.items[0].verdict, null); + + const human = await fleet(["findings", "--app", "cliapp"]); + assert.equal(human.code, 0, human.stderr); + assert.match(human.stdout, /Crashes on Save/); + + const file = path.join(tempDir("fleet-flow-"), "repro.yaml"); + const wrote = await fleet(["finding", String(id), "--flow", file]); + assert.equal(wrote.code, 0, wrote.stderr); + assert.equal(readFileSync(file, "utf8"), flow); + const shown = await fleet(["finding", `#${id}`]); + assert.match(shown.stdout, /1\. Open the app/); + + const early = await fleet(["verdict", String(id), "fixed"]); + assert.equal(early.code, 5, "fixed before real is refused"); + assert.match(early.stderr, /conflict/); + assert.equal((await fleet(["verdict", String(id), "maybe"])).code, 2); + const real = await fleet(["verdict", String(id), "real", "--note", "seen it"]); + assert.equal(real.code, 0, real.stderr); + assert.match(real.stdout, /candidate/); + const fixed = await fleet(["verdict", String(id), "fixed", "--json"]); + assert.equal(fixed.code, 0, fixed.stderr); + assert.equal(json(fixed).finding.regression.state, "active"); + const notABug = await fleet(["verdict", String(id), "not-a-bug"]); + assert.match(notABug.stdout, /retired/); +}); diff --git a/fleet/test/helpers.ts b/fleet/test/helpers.ts index 7387466..5531cc3 100644 --- a/fleet/test/helpers.ts +++ b/fleet/test/helpers.ts @@ -160,3 +160,22 @@ export async function upload(base: string, bytes: Buffer | string, name: string, } export const tempDir = (prefix: string) => mkdtempSync(path.join(tmpdir(), prefix)); + +/** A night-QA finding posted the way the explore workload posts one, with its replay flow uploaded. Returns the id and the flow. */ +export async function postFinding(base: string, app: string, over: Record = {}): Promise<{ id: number; flow: string }> { + const flow = `appId: com.example.${app}\nname: "repro"\n---\n- launchApp:\n clearState: true\n- tapOn:\n point: "50%,90%"\n`; + const sha = await upload(base, flow, "replay.yaml"); + const res = await fetch(`${base}/findings`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + fingerprint: `${app}-${Math.random().toString(36).slice(2)}`, app, build: "1.0 (1)", platform: "android", device_id: "test-phone", + job_id: "explore-test", mission_id: "save", check: "crash", severity: "high", title: "Crashes on Save", detail: "FATAL EXCEPTION", + screen: "s1", screen_name: "Add plant", steps: ["Open the app", "Tap Save"], + replay: { kind: "maestro", sha256: sha, attempts: 2, reproduced: 2 }, artifacts: { replay: sha }, + ...over, + }), + }); + if (!res.ok) throw new Error(`POST /findings ${res.status}: ${await res.text()}`); + return { id: ((await res.json()) as { id: number }).id, flow }; +} diff --git a/fleet/test/mcp.test.ts b/fleet/test/mcp.test.ts index 027ed34..b503c44 100644 --- a/fleet/test/mcp.test.ts +++ b/fleet/test/mcp.test.ts @@ -14,7 +14,7 @@ import { resolveBrain, resolveToken } from "../src/connect.js"; import { handle, type Session } from "../src/mcp/protocol.js"; import { localSession } from "../src/mcp/stdio.js"; import { TOOLS } from "../src/mcp/tools.js"; -import { fakeDevice, FLEET, PNG, startCollector, TSX, upload, type Collector } from "./helpers.js"; +import { fakeDevice, FLEET, PNG, postFinding, startCollector, TSX, upload, type Collector } from "./helpers.js"; let c: Collector; let session: Session; @@ -169,3 +169,33 @@ test("interop: the MCP SDK client drives `fleet mcp` over stdio", async () => { await client.close(); } }); + +test("fleet_findings, fleet_finding with its flow, and fleet_verdict", async () => { + const { id, flow } = await postFinding(c.base, "mcpapp"); + const list = await call("fleet_findings", { app: "mcpapp", since: "24h" }); + assert.ok(!list.isError, list.content[0].text); + assert.equal(list.structuredContent.total, 1); + assert.equal(list.structuredContent.findings[0].id, id); + assert.equal(list.structuredContent.findings[0].replay, "reproduced 2/2"); + + const one = await call("fleet_finding", { id }); + assert.ok(!one.isError, one.content[0].text); + assert.equal(one.structuredContent.replay_flow.text, flow, "a small flow comes back inline"); + assert.deepEqual(one.structuredContent.steps, ["Open the app", "Tap Save"]); + + const early = await call("fleet_verdict", { id, verdict: "fixed" }); + assert.equal(early.isError, true); + assert.match(early.content[0].text ?? "", /conflict/); + assert.match(early.content[0].text ?? "", /real first/); + const real = await call("fleet_verdict", { id, verdict: "real", note: "reproduced locally" }); + assert.equal(real.structuredContent.regression, "candidate"); + const fixed = await call("fleet_verdict", { id, verdict: "fixed", note: "PR #1" }); + assert.equal(fixed.structuredContent.verdict, "fixed"); + assert.equal(fixed.structuredContent.regression, "active"); + + const { result } = await rpc("tools/list"); + const by = Object.fromEntries(result.tools.map((t: { name: string }) => [t.name, t])); + assert.equal(by.fleet_verdict.annotations.readOnlyHint, false); + assert.equal(by.fleet_findings.annotations.readOnlyHint, true); + assert.ok(by.fleet_finding.outputSchema && by.fleet_findings.outputSchema && by.fleet_verdict.outputSchema); +}); diff --git a/skills/fleet-runner/SKILL.md b/skills/fleet-runner/SKILL.md index a43684c..691c863 100644 --- a/skills/fleet-runner/SKILL.md +++ b/skills/fleet-runner/SKILL.md @@ -32,6 +32,9 @@ the shared names. | `fleet_results` | `fleetr results` | read | no | | `fleet_flaky` | `fleetr flaky` | read | no | | `fleet_artifact` | `fleetr artifact SHA [--out FILE]` | read | no | +| `fleet_findings` | `fleetr findings [--app A] [--since 24h]` | read | no | +| `fleet_finding` | `fleetr finding ID [--flow FILE]` | read | no | +| `fleet_verdict` | `fleetr verdict ID VERDICT [--note TEXT]` | run | yes | | `fleet_whoami` | `fleetr whoami` | read | no | @@ -70,6 +73,29 @@ job) or `running`. `fleetr run`, `job`, `wait` and `retry --wait` exit with: - **5** refused or unreachable: no permission, unknown workload or job, or no collector. Report it; do not work around it. +## Reading a finding + +Night QA (the `explore` workload) drives each app overnight and files what +looks wrong as a finding. `fleetr findings --since 24h` (`fleet_findings +since=24h`) is last night; `fleetr finding ID` (`fleet_finding id=ID`) is one +of them with its steps, evidence and replay flow; `--flow FILE` writes the +flow so you can reproduce it. + +- **Read the replay count first.** Every finding was replayed on a clean + install before it was filed: `reproduced 2/2` is solid, `flaky 1/2` is + weaker, `crash log` means the platform's crash log is the evidence. Look at + the screenshot (`fleetr artifact SHA --out DIR`) before you change code. +- **`agent_mistake` is not a bug.** It means the model misread the app and the + app behaved correctly. Do not fix the app for it. `not_a_bug` likewise. +- **Never mark a finding `fixed` without a fix merged.** `fixed` comes only + after `real`, and it puts the finding's flow into the app's nightly as a + regression test that reopens the finding if a crash comes back. Put the + fix's PR in `--note`. If you are unsure whether it is real, leave the + verdict to a person. +- **`seen_count` and `builds_seen`** say how often and on which builds it came + back; a finding open again with a `regressed on build ...` line in its + history is a fix that did not hold. + ## Rules - **Skipped is not passed.** Report it as "did not run" with the reason.