Findings reach the code: /api/v1, fleetr and MCP, and fixed findings become nightly regression tests

CHANGELOG.md

@@ -26,8 +26,8 @@ a CI job fails when they disagree.
   `GITHUB_EVENT_NAME` is `pull_request` (or with `--channel pr`). The pull
   request's own job still names its build by hash. Builds uploaded before this
   count as merged builds, and the same bytes republished from main come off the
-  channel. The `/api/v1` artifact document does not show the channel yet;
-  `GET /api/artifacts` does.
+  channel. The `/api/v1` artifact document shows it as `channel: pr|main`
+  since #96.
 - **`FLEET_PEER_TOKENS`: the brain switcher can read a peer with tokens on.**
   The peers proxy sent no token, so a peer running `FLEET_AUTH=tokens`
   answered 401 through it. Each peer's URL now maps to a `read` token that
@@ -95,6 +95,52 @@ a CI job fails when they disagree.
   pull request, docs-only ones included, when a marked number disagrees with
   the shipped entries in `collector/src/workloads/catalog.ts`. The collector's
   suite checks that reading against the real `CATALOG`. ([#89](https://github.com/addisdev/fleet-runner/pull/89))
+- **Findings are in the control API.** `/api/v1` had 22 routes and none was
+  findings, so a coding agent could run a job but not read the bug list night
+  QA wrote for it. `GET /api/v1/findings` (filter by `app`, `state`,
+  `verdict`, `check`, `since`, which takes `24h` as well as ISO 8601),
+  `GET /api/v1/findings/{id}` (the steps in words, the evidence as links, the
+  replay flow inline up to 16 KiB, the regression test, the verdict history)
+  and `POST /api/v1/findings/{id}/verdict` (scope `run`, audited). The legacy
+  `/api/findings` routes call the same functions and gained the same filters
+  and fields. ([#96](https://github.com/addisdev/fleet-runner/pull/96))
+- **`fleetr findings`, `fleetr finding`, `fleetr verdict`, and three MCP
+  tools.** "What did last night find in GreenFolio, and give me the Maestro
+  flow" is now one call: `fleetr findings --app greenfolio --since 24h`, then
+  `fleetr finding ID --flow repro.yaml`; `fleet_findings`, `fleet_finding`
+  (flow text inline) and `fleet_verdict` (not read-only, scope `run`) do the
+  same over MCP. `fleetr findings --since 24h --json` is the input a scheduled
+  agent session can start from; its shape is documented in docs/agents.md.
+  The skill has a section on reading a finding: the replay count first,
+  `agent_mistake` is not a bug, and never `fixed` without a fix merged. ([#96](https://github.com/addisdev/fleet-runner/pull/96))
+- **A fixed finding becomes a regression test.** A new verdict, `fixed`, comes
+  only after `real` (a 409 otherwise). Marking a finding real copies its replay
+  flow into a new `regressions` table as a candidate; fixed makes it active;
+  duplicate, not a bug or agent's mistake retire it. A nightly `ui-test` with
+  `params.regressions: "<app>"` pulls the app's active flows from the
+  collector at run time and runs each after its own flow, so a finding marked
+  fixed today is in tonight's run without anyone editing a schedule. For a
+  crash or ANR finding on Android the executor reads `logcat -b crash` and
+  `am_anr` around the flow; a new crash of the package is `reproduced`, fails
+  the row and reopens the finding (verdict cleared, seen once more, the build
+  added, a `regressed on build ...` line in its new verdict history). Every
+  other check can only be asserted as "replays without error", and the docs
+  and each run's detail say so; a failing flow fails the row but reopens
+  nothing. GitHub issues are only ever created, never read back, so closing
+  one does not mark a finding fixed. The dashboard has a **Fixed** button
+  (<kbd>f</kbd>) and a regression panel. A database opened by this release
+  rebuilds the findings table once to allow `fixed` and backfills one history
+  row per existing verdict. Not yet run on a real nightly: covered by the
+  suite against fake crash readings and a fake Maestro. ([#96](https://github.com/addisdev/fleet-runner/pull/96))
+- **The Findings page reads and judges through `/api/v1`.** Its list, detail
+  and verdict calls go through the generated client, so its types are the API's
+  and a dashboard verdict is the same call as `fleetr verdict`. `GET
+  /api/findings`, `GET /api/findings/{id}` and `POST
+  /api/findings/{id}/verdict` now carry `Deprecation` and a successor `Link`;
+  `/api/findings/precision`, `/digest` and `/issues` have no v1 counterpart and
+  do not. The v1 list carries `facets` (every app and check, and open/triaged
+  counts) like the device and job lists, and the v1 verdict keeps the
+  `FLEET_DASH_TOKEN` speed bump the older route had. Part of #83. ([#96](https://github.com/addisdev/fleet-runner/pull/96))
 
 ### Changed
 
@@ -195,6 +241,10 @@ a CI job fails when they disagree.
 
 ### Fixed
 
+- **The OpenAPI document named the wrong status for two enqueue refusals.**
+  `POST /api/v1/jobs` listed `no_matching_devices` and `no_build` under 422,
+  but `queue.ts` refuses both with 400. The 400 and 422 entries now name the
+  codes each really carries. ([#96](https://github.com/addisdev/fleet-runner/pull/96))
 - **The older job routes skipped two auth checks.** With `FLEET_AUTH=tokens`,
   `POST /api/jobs/:id/cancel` and `/retry` let any `run` token stop or re-run
   anyone's job, and `POST /jobs` and `POST /api/jobs` accepted the `shell`

collector/dash/src/api.ts

@@ -230,69 +230,12 @@ export type Health = {
 
 // --- night-QA findings (mirrors src/findings.ts) ---
 
-export type FindingVerdict = "real" | "duplicate" | "not_a_bug" | "agent_mistake";
-
-export type FindingIssue = {
-  state: "dry_run" | "pending" | "filed" | "failed" | "capped" | "no_repo";
-  /** True means nothing was sent to GitHub. */
-  dry_run: boolean;
-  repo: string | null;
-  title: string;
-  body: string;
-  labels: string[];
-  url: string | null;
-  number: number | null;
-  detail: string;
-  at: string;
-};
-
-export type Finding = {
-  id: number;
-  fingerprint: string;
-  app: string;
-  build: string;
-  platform: string;
-  device_id: string;
-  job_id: string;
-  last_job_id: string | null;
-  mission_id: string;
-  /** An open string, like Device.platform: the workload gains checks before the dashboard learns their names. */
-  check: string;
-  severity: "high" | "medium" | "low";
-  title: string;
-  detail: string;
-  screen: string;
-  screen_name: string;
-  steps: string[];
-  replay: { kind: string; sha256: string | null; attempts: number; reproduced: number } | null;
-  /** "reproduced 2/2", "flaky 1/2", "crash log". */
-  replay_label: string;
-  artifacts: { shot?: string; sheet?: string; trajectory?: string; log?: string; replay?: string };
-  first_seen: string | null;
-  last_seen: string | null;
-  seen_count: number;
-  builds_seen: string[];
-  verdict: FindingVerdict | null;
-  verdict_note: string | null;
-  verdict_at: string | null;
-  duplicate_of: number | null;
-  issue: FindingIssue | null;
-  status: "open" | "triaged";
-};
-
-export type FindingDetail = Finding & {
-  duplicates: { id: number; title: string; seen_count: number }[];
-  duplicate_of_title: string | null;
-};
-
-export type FindingList = {
-  findings: Finding[];
-  counts: { open: number; triaged: number; total: number };
-  /** Facets over every finding, not the filtered set. */
-  apps: string[];
-  checks: string[];
-  limit: number;
-};
+// Findings, one finding and the verdict come from /api/v1, so their types are
+// the generated client's. The precision table is not in /api/v1 and keeps its
+// own type below.
+export type { Finding, FindingDetail, FindingList } from "./client.js";
+import type { Finding as V1Finding } from "./client.js";
+export type FindingVerdict = NonNullable<V1Finding["verdict"]>;
 
 export type FindingPrecision = {
   floor: number;

collector/dash/src/keys.ts

@@ -21,7 +21,7 @@ const GOTO: Record<string, [string, string]> = {
 export const SHORTCUTS = [
   ...Object.entries(GOTO).map(([k, [, label]]) => ({ keys: `g ${k}`, does: label })),
   { keys: "/", does: "Focus the search box on this screen" },
-  { keys: "r d n a", does: "On a finding: Real, Duplicate, Not a bug, Agent's mistake" },
+  { keys: "r d n a f", does: "On a finding: Real, Duplicate, Not a bug, Agent's mistake, Fixed (after Real)" },
   { keys: "?", does: "Show this help" },
   { keys: "Esc", does: "Close help, or leave the search box" },
 ];

collector/dash/src/pages/Findings.tsx

@@ -10,8 +10,9 @@
 // is switched off for that app (the precision table at the top), so a wrong
 // verdict costs tomorrow night something, and the buttons say what they mean.
 import { useEffect, useRef, useState } from "preact/hooks";
-import { useApi, type Finding, type FindingDetail as Detail, type FindingList, type FindingPrecision, type FindingVerdict } from "../api.js";
-import { mutate, useMutation } from "../mutate.js";
+import { useApi, useLoad, type Finding, type FindingDetail as Detail, type FindingPrecision, type FindingVerdict } from "../api.js";
+import { change, fleet, read } from "../client.js";
+import { useMutation } from "../mutate.js";
 import { useDeviceNames } from "../names.js";
 import { navigate, useQuery } from "../router.js";
 import {
@@ -28,13 +29,15 @@ export const VERDICT_LABEL: Record<FindingVerdict, string> = {
   duplicate: "Duplicate",
   not_a_bug: "Not a bug",
   agent_mistake: "Agent's mistake",
+  fixed: "Fixed",
 };
 
 const VERDICT_PILL: Record<FindingVerdict, string> = {
   real: "failed",
   duplicate: "queued",
   not_a_bug: "cancelled",
   agent_mistake: "cancelled",
+  fixed: "done",
 };
 
 /** reproduced N/N is solid, flaky is a warning, a crash log is its own evidence. */
@@ -166,10 +169,12 @@ export function Findings() {
   const app = q.get("app") ?? "";
   const check = q.get("check") ?? "";
   const status = (STATUSES as readonly string[]).includes(q.get("status") ?? "") ? (q.get("status") as string) : "open";
-  const params = new URLSearchParams({ status });
-  if (app) params.set("app", app);
-  if (check) params.set("check", check);
-  const state = useApi<FindingList>(`/api/findings?${params}`, ["finding"], 60_000);
+  const state = useLoad(
+    `findings ${status} ${app} ${check}`,
+    () => read(fleet.listFindings({ state: status as "open" | "triaged" | "all", app: app || undefined, check: check || undefined, per_page: 100 })),
+    ["finding"],
+    60_000,
+  );
 
   return (
     <>
@@ -178,7 +183,7 @@ export function Findings() {
       <Loaded state={state} what="findings">
         {(d) => (
           <Panel
-            title={`${d.counts.open} open · ${d.counts.triaged} triaged`}
+            title={`${d.facets.counts.open} open · ${d.facets.counts.triaged} triaged`}
             aside={
               <span class="windows">
                 {STATUSES.map((s) => (
@@ -191,19 +196,19 @@ export function Findings() {
             }
           >
             <Filters active={!!(app || check)} onClear={() => setQuery({ app: null, check: null })}>
-              <Select label="app" value={app} options={d.apps} onChange={(v) => setQuery({ app: v })} />
-              <Select label="check" value={check} options={d.checks} onChange={(v) => setQuery({ check: v })} />
+              <Select label="app" value={app} options={d.facets.apps} onChange={(v) => setQuery({ app: v })} />
+              <Select label="check" value={check} options={d.facets.checks} onChange={(v) => setQuery({ check: v })} />
             </Filters>
-            {d.findings.length === 0 ? (
+            {d.items.length === 0 ? (
               <p class="empty">
-                {d.counts.total === 0
+                {d.facets.counts.total === 0
                   ? "Nothing found yet. Night QA posts here after it has replayed a finding on a clean install."
                   : status === "open"
                     ? "Nothing waiting for a verdict."
                     : "No findings match."}
               </p>
             ) : (
-              <FindingRows findings={d.findings} />
+              <FindingRows findings={d.items} />
             )}
           </Panel>
         )}
@@ -231,7 +236,9 @@ const typing = (t: EventTarget | null) => {
 /**
  * The four buttons, the note, and the duplicate picker.
  *
- * r / d / n / a do the same as the buttons. They are ignored straight after a
+ * r / d / n / a / f do the same as the buttons. Fixed is offered only once a
+ * finding is real (the collector refuses it otherwise): it makes the finding's
+ * replay flow a regression test in the app's nightly. They are ignored straight after a
  * `g`, because "g r" is already "go to Results" and a shortcut that filed a
  * verdict on the way out of the page would be a nasty surprise.
  */
@@ -240,19 +247,21 @@ function VerdictBox({ f, onDone }: { f: Detail; onDone: () => void }) {
   const [picking, setPicking] = useState(false);
   const [dupOf, setDupOf] = useState("");
   const pickRef = useRef<HTMLSelectElement>(null);
-  const others = useApi<FindingList>(picking ? `/api/findings?app=${encodeURIComponent(f.app)}&status=all&limit=200` : null);
+  const others = useLoad(picking ? `findings of ${f.app}` : null, () => read(fleet.listFindings({ app: f.app, state: "all", per_page: 200 })));
 
   const [pending, setPending] = useState<FindingVerdict | null>(null);
+  // Fixed follows real, or a regression test that brought the finding back.
+  const canFix = f.verdict === "real" || f.verdict === "fixed" || f.regression?.state === "active";
   const send = useMutation(async () => {
     const body: Record<string, unknown> = { verdict: pending, note: note.trim() || null };
     if (pending === "duplicate") body.duplicate_of = Number(dupOf);
-    const r = await mutate("POST", `/api/findings/${f.id}/verdict`, body);
+    const r = await change(fleet.setFindingVerdict(String(f.id), body as Parameters<typeof fleet.setFindingVerdict>[1]));
     setPicking(false);
     onDone();
     return r;
   });
   const reopen = useMutation(async () => {
-    const r = await mutate("POST", `/api/findings/${f.id}/verdict`, { verdict: null });
+    const r = await change(fleet.setFindingVerdict(String(f.id), { verdict: null }));
     onDone();
     return r;
   });
@@ -288,7 +297,8 @@ function VerdictBox({ f, onDone }: { f: Detail; onDone: () => void }) {
         afterG = false;
         return;
       }
-      const v = ({ r: "real", d: "duplicate", n: "not_a_bug", a: "agent_mistake" } as Record<string, FindingVerdict>)[e.key];
+      const v = ({ r: "real", d: "duplicate", n: "not_a_bug", a: "agent_mistake", f: "fixed" } as Record<string, FindingVerdict>)[e.key];
+      if (v === "fixed" && !canFix) return;
       if (v) {
         e.preventDefault();
         judge(v);
@@ -299,9 +309,9 @@ function VerdictBox({ f, onDone }: { f: Detail; onDone: () => void }) {
       removeEventListener("keydown", onKey);
       clearTimeout(timer);
     };
-  }, [f.id, note]);
+  }, [f.id, note, canFix]);
 
-  const candidates = (others.data?.findings ?? []).filter((o) => o.id !== f.id);
+  const candidates = (others.data?.items ?? []).filter((o) => o.id !== f.id);
   return (
     <Panel
       title="Verdict"
@@ -310,20 +320,22 @@ function VerdictBox({ f, onDone }: { f: Detail; onDone: () => void }) {
           {VERDICT_LABEL[f.verdict]} · {clock(f.verdict_at)}{" "}
           <button type="button" class="linkish" onClick={() => void reopen.go()}>reopen</button>
         </span>
-      ) : <span class="faint">keys: r d n a</span>}
+      ) : <span class="faint">keys: r d n a f</span>}
     >
       <div class="verdicts">
         {(Object.keys(VERDICT_LABEL) as FindingVerdict[]).map((v) => (
           <Button
             key={v}
             tone={f.verdict === v ? "primary" : undefined}
             busy={send.busy && pending === v}
+            disabled={v === "fixed" && !canFix}
             onClick={() => judge(v)}
             title={{
               real: "A bug in the app. Counts towards this check's precision.",
               duplicate: "The same problem as another finding. Left out of precision.",
               not_a_bug: "The app is fine; the check was wrong. Counts against this check.",
               agent_mistake: "The model did something silly and the app reacted correctly. Counts against this check.",
+              fixed: "Real, and a fix has merged. Its replay flow joins the app's nightly ui-test as a regression test. Only after Real.",
             }[v]}
           >
             <kbd>{v === "not_a_bug" ? "n" : v[0]}</kbd> {VERDICT_LABEL[v]}
@@ -361,8 +373,8 @@ function VerdictBox({ f, onDone }: { f: Detail; onDone: () => void }) {
 
 /** The next finding still waiting for a verdict, so a morning's triage is a run of keystrokes. */
 function NextOpen({ f }: { f: Detail }) {
-  const list = useApi<FindingList>(`/api/findings?status=open&limit=200`, ["finding"]);
-  const next = list.data?.findings.find((o) => o.id !== f.id);
+  const list = useLoad("findings open", () => read(fleet.listFindings({ state: "open", per_page: 200 })), ["finding"]);
+  const next = list.data?.items.find((o) => o.id !== f.id);
   if (!list.data) return null;
   return next ? (
     <Link to={`/findings/${next.id}`} class="next-open">next open: {next.title} →</Link>
@@ -398,8 +410,44 @@ function IssuePanel({ issue }: { issue: NonNullable<Finding["issue"]> }) {
   );
 }
 
+const REGRESSION_LABEL: Record<NonNullable<Detail["regression"]>["state"], string> = {
+  candidate: "candidate — joins the nightly when marked fixed",
+  active: "active — in the nightly",
+  retired: "retired",
+};
+
+/** The regression test a real finding became, and the verdict history that moved it. */
+function RegressionPanel({ r, history }: { r: NonNullable<Detail["regression"]>; history: Detail["history"] }) {
+  return (
+    <Panel
+      title="Regression test"
+      aside={<Pill kind={r.state === "active" ? "done" : r.state === "candidate" ? "queued" : "cancelled"}>{REGRESSION_LABEL[r.state]}</Pill>}
+    >
+      <p class="empty">
+        {r.kind} flow · {r.asserts === "no_crash" ? "asserts the app does not crash during it" : "asserts only that it replays without error"}
+        {" "}· run {r.runs}×
+        {r.last_outcome && <> · last {r.last_outcome}{r.last_run_at ? ` ${agoFrom(r.last_run_at)}` : ""}</>}
+        {r.last_job_id && <> in <Link to={`/jobs/${encodeURIComponent(r.last_job_id)}`}>{r.last_job_id}</Link></>}
+      </p>
+      {r.last_detail && <p class="stub">{r.last_detail}</p>}
+      {history.length > 0 && (
+        <ul class="dup-list">
+          {history.map((h, i) => (
+            <li key={i}>
+              <span class="faint">{clock(h.at)}</span>{" "}
+              {h.event === "regressed" ? "regressed" : h.verdict ? VERDICT_LABEL[h.verdict] : "reopened"}
+              {h.actor && <span class="faint"> · {h.actor}</span>}
+              {h.note && <> — {h.note}</>}
+            </li>
+          ))}
+        </ul>
+      )}
+    </Panel>
+  );
+}
+
 export function FindingPage({ id }: { id: string }) {
-  const state = useApi<Detail>(`/api/findings/${encodeURIComponent(id)}`, ["finding"]);
+  const state = useLoad(`finding ${id}`, () => read(fleet.getFinding(id)), ["finding"]);
   const names = useDeviceNames();
 
   return (
@@ -494,6 +542,8 @@ export function FindingPage({ id }: { id: string }) {
               </Panel>
             )}
 
+            {f.regression && <RegressionPanel r={f.regression} history={f.history} />}
+
             {f.issue && <IssuePanel issue={f.issue} />}
 
             <Actions>